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Executive  Summary 


Background 

The  Predictable  Assembly  from  Certifiable  Components  (PACC)  Initiative  at  the  Software 
Engineering  Institute  (SEI^'^^)  is  developing  methods  and  technologies  for  predictable  assem¬ 
bly.  A  software  development  activity  that  builds  systems  from  components  is  predictable  if  the 
runtime  behavior  of  an  assembly  of  components  can  be  predicted  from  known  properties  of 
components  and  their  patterns  of  interactions  (connections),  and  if  these  predictions  can  be 
objectively  validated.  A  component  is  certifiable  if  these  known  properties  can  be  obtained  or 
validated  by  independent  third  parties. 

The  SEI’s  approach  to  predictable  assembly^  is  through  prediction-enabled  component  tech¬ 
nology  (PECT).  At  the  highest  level,  PECT  is  a  scheme  for  systematic  and  repeatable  integra¬ 
tion  of  software  component  technology,  software  architecture  technology,  and  design  analysis 
and  verification  technology.  This  scheme  is  not  simply  technological;  it  also  includes  the  pro¬ 
cesses  needed  to  design  and  validate  the  predictive  powers  of  a  PECT.  A  PECT,  then,  is  a 
packaging  of  engineering  methods  and  a  supporting  technical  infrastructure  that,  together, 
enable  predictable  assembly  from  certifiable  components. 

PACC  became  an  SEI  initiative  on  October  1, 2002.  Prior  to  that,  from  1999-2001,  PACC  was 
considered  an  exploratory  research  project.  The  objective  of  such  a  project  at  the  SEI  is  to 
develop  an  understanding  of  a  problem  area  and  proposed  solutions  for  it.  During  the  third 
year  of  exploratory  research  (2001-2002)  the  SEI,  in  partnership  with  an  industrial  sponsor, 
the  Asea  Brown  Boveri,  Ltd.  Corporate  Research  Center  (ABB/CRC),  undertook  the  prototyp¬ 
ing  effort  described  in  this  report. 


Objective 

The  objective  of  this  prototyping  effort  was  not  to  develop  a  PECT  solution  for  a  particular 
design  problem  per  se,  but  rather  to  explore  as  many  aspects  of  PECT  as  is  practical  within  a 
limited  time  frame.  Our  primary  concerns  were  methodological  and  practical:  Could  we  define 
a  process  for  developing,  and  validating  PECTs?  Would  the  process  be  practical?  What  are  the 
risks  to  transitioning  PECT  to  practice?  What  are  the  technical  limits  of  PECT,  and  how  might 


1 .  SEI  is  a  service  mark  of  Carnegie  Mellon  University. 

2.  In  this  report,  the  term  predictable  assembly  means  predictable  assembly  from  certifiable  components. 
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they  be  overcome?  On  more  than  one  occasion,  our  prototyping  effort  was  diverted  to  explore 
these  and  similar  questions. 


Approach 

To  focus  and  ground  our  research,  we  selected  as  a  prototype  problem  the  substation  automa¬ 
tion  system  (SAS),  an  application  area  in  the  domain  of  power  generation,  transmission,  and 
management.  This  problem  area  was  chosen  because  it  is  well  bounded,  well  defined,  and  rep¬ 
resentative  of  the  broader  class  of  critical  infrastructure  systems.  The  SEI  defined  three  model 
problems  in  the  predictable  assembly  of  SAS:  the  assembly  of  an  operator  station,  a  primary 
equipment  controller,  and  an  integrated  operator/controller.  The  nominal  objective  of  the  pro¬ 
totype  was  to  develop  three  PECTs,  one  for  each  model  problem. 


Results  of  This  Work 

This  report  describes  the  results  of  an  experimental  application  of  PECT  to  SAS,  focusing  on 
the  primary  equipment  PECT,  the  SAS  switch  controller.  A  low-power,  high-speed  switch  was 
developed  by  the  SEI  to  simulate  primary  equipment.  Controller  assemblies  were  developed  in 
accordance  with  the  International  Electrotechnical  Commission  (lEC)  61850  standard  for  sub¬ 
station  controllers,  and  laboratory  scenarios  were  developed  for  external  switch  control  and 
over-current  protection.  A  family  of  latency  models  and  their  interpretations  (k*)  were  devel¬ 
oped  to  predict  point-to-point,  intra-assembly  (intra-controller)  latency.  One  member  of  this 
family,  average  case  latency  with  blocking  and  asynchrony  empirically  validated. 

Primary  results.  The  primary  results  of  this  work  were  methodological.  We  developed  an 
overall  process  model  for  the  design,  development,  and  validation  of  PECTs.  We  studied  two 
key  processes  in  depth:  co-refinement  and  empirical  validation.  Technically,  co-refinement  is 
the  process  of  defining  an  interpretation  from  component  model  to  analysis  model.  Intuitively, 
it  is  a  negotiation  that  results  in  a  component  model  that  is  expressive  enough  to  span  an 
intended  application  area,  but  restrictive  enough  to  ensure  that  analysis  is  tractable.  Techni¬ 
cally,  empirical  validation  defines  measurement  procedures  for  obtaining  component  mea¬ 
sures  and  assembly  measures,  and  for  designing  the  experiments  to  compare  the  predicted  and 
observed  assembly  measures.  Intuitively,  it  is  a  means  of  attaching  meaningful  statistical 
labels  to  components  and  the  predictive  powers  of  PECT  analysis. 

Secondary  results.  The  secondary  results  of  this  work  were  technological.  We  developed  a 
measurement  and  validation  infrastructure  and  gained  valuable  experience  in  the  construction 
of  a  laboratory  environment  for  empirical  validation.  More  significantly,  we  specified  an  ana¬ 
lytically  extensible  component  model.  This  model  defines  the  basics  of  component  interface 
and  connection  topology  (assembly)  and  is  extended  by  formal  interpretations  to  one  or  more 
analysis  models.  An  analysis  model  supports  compositional  reasoning  about  assembly  proper¬ 
ties  and  defines  the  component  properties  required  to  predict  Each  interpretation 
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may  impose  additional  constraints  on  components  and  their  topologies.  In  return,  each  analy¬ 
sis  model  guarantees  that  assemblies  of  components  satisfying  these  constraints  will  be  ana- 
lyzable,  and  hence  be  predictable  by  construction,  with  respect  to 

Although  the  work  emphasized  empirical  predictability,/omia/  approaches  to  predictability 
were  also  explored.  Specifically,  safety  conditions  for  the  prototype  (hardware)  switch  and 
lEC  61850  (software)  controller  dealing  with  the  over-current  protection  components  were 
expressed  in  branching  time  temporal  logic.  State  machines  for  these  components  were  devel¬ 
oped,  and  the  safety  and  liveness  conditions  were  verified  using  the  SMV  model  checker.^ 

Tertiary  results.  The  tertiary  results  of  this  work  were  the  PECTs  themselves.  A  PECT  was 
developed  for  the  operator  interface  and  controller.  The  operator  PECT  was  developed  on  the 
Microsoft  -NET  environment.  Latency  predictions  for  operator  interfaces  were  accurate  (<  3% 
magnitude  of  relative  error  [MRE])  but  not  technically  demanding.  More  interesting  was  the 
controller  PECT,  which  was  substantially  more  complex  and  demanding.  Controller  assem¬ 
blies  introduced  threading,  contention,  priority-based  scheduling,  and  periodicity  not  encoun¬ 
tered  in  the  operator  PECT.  The  original  normative  requirement  for  controller  latency 
prediction  was  a  90%  confidence  interval  that  80%  of  predictions  would  not  exceed  an  upper 
bound  5%  MRE.  This  norm  was  arbitrary  and  served  primarily  as  a  plot  device.  Still,  ^aba 
predictions  came  within  a  whisker  of  satisfying  that  norm  despite  the  use  of  a  non-real-time, 
non-optimized  operating  system,  Microsoft  Windows  2000:  we  achieved  a  >99%  confidence 
interval  that  80%  of  predictions  will  fall  within  an  upper  bound  of  10%  MRE.  Moreover, 
under  the  assumption  that  identified  systematic  error  can  be  eliminated  (reasonable,  given  the 
stability  of  the  error),  these  results  will  improve  to  a  >99%  confidence  interval  that  80%  of 
predictions  will  fall  within  an  upper  bound  of  4%  MRE. 


Next  Steps 

The  results  of  the  exploratory  prototype  are  encouraging.  However,  as  expected,  there  are 
aspects  of  PECT  that  require  further  elaboration,  and  some  that  have  yet  to  be  explored.  This 
report  identifies  two  areas  requiring  attention. 

First,  a  technology  infrastructure  for  specifying  and  deploying  components  and  their  assem¬ 
blies  must  be  constructed;  the  substation  automation  prototypes  relied  on  manual  processes  for 
much  of  this.  We  developed  a  substantial  suite  of  tools  to  support  empirical  validation,  but 
greater  automation  is  still  required  to  generate  the  massive  data  sets  required  to  adequately 
validate  even  moderately  complex  analysis  models. 


3.  SMV  is  a  symbolic  model-checking  tool  developed  at  Carnegie  Mellon  University.  For  more  information  about  it,  see 
<http://www.cs.cmu.edu/~modelcheck/smv/smvmanual.ps>. 
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Second,  the  methods  and  technologies  needed  to  support  independent  (third-party)  measure¬ 
ment  and  certification  of  components  and  PECTs  must  be  established;  the  substation  automa¬ 
tion  prototypes  relied  on  in  situ  measurements.  Means  must  be  established  for  acquiring 
measurements  in  third-party  environments,  establishing  and  maintaining  trust  in  assertions 
about  those  measurements,  and  extrapolating  those  assertions  to  different,  possibly  heteroge¬ 
neous  development  and  deployment  environments. 
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Abstract 


The  Predictable  Assembly  from  Certifiable  Components  (PACC)  Initiative  at  the  Software 
Engineering  Institute  (SEI^*^)  is  developing  methods  and  technologies  for  predictable  assem¬ 
bly.  A  software  development  activity  that  builds  systems  from  components  is  predictable  if  the 
runtime  behavior  of  an  assembly  of  components  can  be  predicted  from  known  properties  of 
components  and  their  patterns  of  interactions  (connections),  and  if  these  predictions  can  be 
objectively  validated.  A  component  is  certifiable  if  these  known  properties  can  be  obtained  or 
validated  by  independent  third  parties.  The  SEFs  technical  approach  to  PACC  rests  on  predic¬ 
tion-enabled  component  technology  (PECT).  At  the  highest  level,  PECT  is  a  scheme  for  sys¬ 
tematic  and  repeatable  integration  of  software  component  technology,  software  architecture 
technology,  and  design  analysis  and  verification  technology.  This  report  describes  the  results 
of  an  exploratory  PECT  prototype  for  substation  automation,  an  application  area  in  the  domain 
of  power  generation,  transmission,  and  management.  This  report  focuses  primarily  on  the 
methodological  aspects  of  PECT;  the  prototype  itself  was  only  a  means  to  expose  and  illustrate 
the  PECT  method. 


4.  SEl  is  a  service  mark  of  Carnegie  Mellon  University. 
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1  Introduction 


The  Predictable  Assembly  from  Certifiable  Components  (PACC)^  Initiative  at  the  Software 
Engineering  Institute  (SEI^^)  is  developing  methods  and  technologies  for  predictable  assem¬ 
bly.  A  software  development  activity  that  builds  systems  from  components  is  predictable  if  the 
runtime  behavior  of  an  assembly  of  components  can  be  predicted  from  known  properties  of 
components  and  their  patterns  of  interactions  (connections),  and  if  these  predictions  can  be 
objectively  validated.  A  component  is  certifiable  if  these  known  properties  can  be  obtained  or 
validated  by  independent  third  parties. 

The  SEI  is  not  alone  in  pursuing  the  objectives  of  PACC,  although  academic  and  commercial 
terminology  may  obscure  this  fact  for  those  not  deeply  familiar  with  the  subject.  The  SEI  tech¬ 
nical  approach  to  PACC  rests  on  prediction-enabled  component  technology  (PECT).  At  the 
highest  level,  PECT  is  a  scheme  for  systematic  and  repeatable  integration  of  software  compo¬ 
nent  technology,  software  architecture  technology,  and  design  analysis  and  verification  tech¬ 
nology.  The  results  of  this  integration  are  engineering  methods  and  a  supporting  technical 
infrastructure  that,  together,  enable  predictable  assembly  from  certifiable  components.^ 

This  report  describes  an  exploratory  prototype  of  PECT.  We  emphasize  the  word  exploratory, 
since  the  objective  of  the  work  was  to  touch  on  as  many  methodological  and  technological 
aspects  of  PECT  as  possible,  using  the  broad  constraints  imposed  by  an  industrially  significant 
problem  area  as  a  guide.  We  selected  the  problem  area  of  substation  automation,  an  application 
area  in  the  domain  of  power  generation,  transmission,  and  management,  because  it  is  well 
bounded,  well  defined,  and  representative  of  the  broader  class  of  critical  infrastructure  sys¬ 
tems. 


5.  For  more  information  on  this  initiative,  see  <http://www.sei.cmu.eclu/pacc>. 

6.  SEI  is  a  service  mark  of  Carnegie  Mellon  University. 

7.  We  will  henceforth  use  the  term  predictable  assembly  to  mean  predictable  assembly  from  certifiable  components. 
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1.1  Background 

In  2000,  the  SEI  initiated  a  feasibility  study  in  predictable  assembly.  The  study  was  under¬ 
taken  to 

•  identify  the  fundamental  science  and  engineering  challenges  posed  by  predictable  assem¬ 
bly  and  software  component  certification 

•  determine  if  the  need  to  address  these  challenges  was  widespread  and  of  economic  or  stra¬ 
tegic  interest  to  the  U.S.  Department  of  Defense  (DoD)  and  to  the  software  industry  as  a 
whole 

•  ascertain  whether  technological  and  methodological  elements  to  address  these  challenges 
existed,  or  could  be  created  with  reasonable  probability  and  effort 

In  2001,  the  SEI  formed  a  collaboration  with  the  Asea  Brown  Boveri,  Ltd.  Corporate  Research 
Center  (ABB/CRC).  This  collaboration  was  established  to  undertake  a  two-year  feasibility 
study  of  predictable  assembly  in  an  industrial  setting.  The  application  area  was  in  power  gen¬ 
eration  and  transmission  systems,  specifically,  substation  automation  systems  (SASs).  Typi¬ 
cally,  an  SAS  is  implemented  as  a  distributed  system  composed  of  20  to  100  computing 
elements  (personal  computers  and/or  other  electronic  computing  devices)  executing  a  mix  of 
soft-  and  hard-real-time  applications.  An  SAS  is  characterized  by  stringent  performance  and 
reliability  requirements. 


1 .2  Approach 

The  SEI/ABB  joint  feasibility  study  intended  to  address  two  broad  feasibility  questions: 

1 .  Can  a  practical  technology  infrastructure  be  developed  to  automate  significant  aspects  of 
predictable  assembly? 

2.  Can  an  engineering  method  be  developed  as  a  basis  for  the  systematic  improvement  of  an 
industry-wide  capability  in  predictable  assembly? 

To  conduct  this  feasibility  exploration,  the  SEI  and  ABB  defined  a  series  of  model  problems. 
Briefly,  a  model  problem  is  a  simplification  of  a  more  complex  one  such  that  a  solution  to  it 
can  be  extrapolated  to  that  of  a  real  problem.  In  2001  and  2002,  three  model  SAS  problems 
were  specified;  in  2002  and  2003,  they  will  be  extended  to  include  the  domain  of  industrial 
robotics. 


1 .3  Objective  of  This  Report 

This  report  consolidates  the  results  and  discoveries  of  the  first  year  of  the  SEI/ABB  investiga¬ 
tion  in  predictable  assembly.  It  is,  in  effect,  a  retrospective  on  an  extended  laboratory  experi- 
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merit.  Our  intent  is  that  this  report  serve  as  a  basis  for  continued  research  within  the  SEI’s 
PACC  Initiative.  Our  intent  is  also  that  this  report,  when  combined  with  related  SEI  publica¬ 
tions  on  predictable  assembly  and  PECT,  will  provide  a  valuable  resource  for  other  applied 
research  in  this  field. 


1 .4  Audience  for  This  Report 

The  audience  for  this  report  is  computer  scientists  and  software  engineering  researchers  with 
an  interest  in  predictable  assembly  from  certifiable  components  (by  any  other  name).  We 
assume  that  readers  have  technical  backgrounds  that  are  both  diverse  and  deep,  spanning  areas 
of  software  architecture,  software  component  technology,  probability  and  measurement  the¬ 
ory,  real-time  systems,  and  model  checking,  to  name  the  major  areas  reflected  in  this  report. 
Therefore,  this  report  is  not  meant  to  serve  as  a  general  introduction  to  predictable  assembly, 
component  technology,  or  PECT. 


1.5  Structure  of  This  Report 

An  overview  of  substation  automation  and  the  details  of  the  SAS  model  problems  are 
described  in  Chapter  2.  In  Chapter  3,  we  provide  an  overview  of  the  technical  concepts  of 
PECT.  The  key  workflows  of  a  process  for  developing  and  validating  a  PECT  are  summarized 
in  Chapter  4.  In  Chapter  5,  we  summarize  the  chronology  of  a  key  development  activity  in  the 
design  of  a  PECT,  called  co-refinement,  the  result  of  which  was  a  controller  PECT  for  predict¬ 
ing  point-to-point  latency  for  any  controller  assembly.  The  techniques  used  to  empirically  val¬ 
idate  the  SAS -controller  PECT  are  outlined  in  Chapter  6.  In  Chapter  7,  the  focus  shifts  from 
empirical  to  formal  approaches  to  predictable  assembly,  where  we  describe  the  use  of  model 
checking  to  verify  safety  and  liveness  conditions  of  controller  assemblies  using  temporal 
logic.  Chapter  8  extracts  the  key  results  of  the  work  to  date,  emphasizing  the  benefits  and  risks 
of  the  PECT  approach  to  predictable  assembly.  The  plan  for  the  next  phase  of  our  investigation 
is  outlined  in  Chapter  9. 

More  details  on  the  work  are  provided  in  appendices  of  this  report.  Appendix  A  describes  the 
Xaba  analysis  model  for  predicting  controller  latency  and  its  interpretation;  this  is  the  model 
that  emerged  from  co-refinement.  A  detailed  account  of  the  empirical  validation  of  ^aba 
provided  in  Appendix  B.  In  Appendix  C,  we  provide  the  temporal-logic  specifications  and 
state  model  used  to  reason  about  controller  safety  conditions.  Appendix  D  provides  a  sche¬ 
matic  for  the  switch  hardware  developed  for  the  prototype.  The  Acronym  List  on  page  133 
defines  the  acronyms  used  in  this  report. 
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2  Model  Problems  for  Substation 
Automation 


The  following  description  of  an  SAS  is  excerpted  from  a  paper  by  Preiss  and  Wegmann  [Preiss 
01]: 


Networks  (power  grids)  of  different  topologies  are  responsible  to  transport 
energy  over  short  or  long  distances  and  finally  distribute  it  to  end-consumers 
(such  as  households  and  companies).  The  nodes  in  such  a  network  are  called 
substations ...  Substations  may  be  manned  or  unmanned  depending  on  the 
importance  of  the  station  and  also  on  its  degree  of  automation.  Substations  are 
controlled  by  Substation  Automation  Systems  (SAS).  Since  unplanned  network 
outages  can  be  disastrous,  an  SAS  is  composed  of  all  the  electronic  equipment 
that  is  needed  to  continuously  control,  monitor,  and  protect  the  network.  This 
covers  all  the  high  voltage  equipment  outside  the  substation  (overhead  lines, 
cables,  etc.)  as  well  as  those  inside  the  substation  (transformers,  circuit  break¬ 
ers,  etc.). 

This  combination  of  challenging  and  concrete  requirements  means  that  an  SAS  provides  a 
good  basis  for  model  problems  in  predictable  assembly.  We  outline  the  basic  structure  of  the 
SAS  model  problems  in  Section  2.1.  Background  information  on  the  International  Electrotech¬ 
nical  Commission  (lEC)  61850  standard,  the  SAS  domain  model  used  to  develop  the  control¬ 
ler  PECT  prototype,  is  provided  in  Section  2.2.  An  overview  of  the  hardware  and  software 
used  in  the  operation  and  controller  prototypes  is  provided  in  Section  2.3. 

2.1  Three  Model  Problems  and  Three  PECTs 

The  scale  of  an  SAS  depends  on  the  cost,  size,  and  criticality  of  the  primary  equipment  that  it 
manages.  For  the  model  problems  described  in  this  report,  we  envisage  a  manned  SAS  that  has 
an  operator  workstation  and  one  or  more  controllers,  each  of  which  controls  some  primary 
equipment,  for  example,  breakers,  switches,  and  transformers.  The  operator  has  a  graphical 
display  console  for  observing  and  interacting  with  primary  equipment.  For  the  purpose  of  this 
report,  we  are  not  concerned  with  details  such  as  the  allocation  of  controllers  to  bays  or  the 
interaction  of  multiple  controllers  (e.g.,  bay  interlocking).  Instead,  we  assume  a  single  opera¬ 
tor  console,  a  single  primary  switch,  and  a  single  controller  for  that  switch.  From  this  general 
structure,  we  defined  three  model  problems,  each  leading  to  a  PECT  as  its  model  solution. 
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I  The  overall  scheme  is  depicted  in  Figure  1.  The  SEI  developed  a  low-power,  high-speed 
switch  (labeled  “SEI  Switch”  in  Figure  1)  to  play  the  role  of  primary  equipment.  One  model 
problem  was  to  develop  a  PECT  for  predictably  assembling  SEI  switch  controllers,  a  second 
model  problem  was  to  develop  a  PECT  for  predictably  assembling  operator  interfaces,  and  a 
third  model  problem  was  to  develop  a  higher  order  PECT  for  composing  controller  and  opera¬ 
tor  assemblies  into  an  overall  SAS  application.  This  PECT  would  allow  operators  to  monitor 
and  control  the  state  of  the  switch. 


Figure  1:  Three  PECTs 

The  PECTs  intended  to  enable  the  prediction  of  the  end-to-end  latency  of  operations  within  an 
operator  assembly,  within  a  controller  assembly,  and  over  a  combined  operator/controller 
assembly.  Nominal  requirements  were  established  for  the  accuracy  and  reliability  of  predic- 

o 

tions  and  were  expressed  as  statistical  tolerance  intervals: 

•  controller  PECT:  99%  confidence  that  80%  of  latency  predictions  will  have  a  magnitude 
of  relative  error  (MRE)  <  5% 

•  operator  PECT:  95%  confidence  that  80%  of  latency  predictions  will  have  MRE  <  10% 

•  SAS  PECT:  95%  confidence  that  80%  of  latency  predictions  will  have  MRE  <  10% 

It  must  be  emphasized  that  these  requirements  were  not  selected  with  any  consideration  for  the 
practicality  of  or  resemblance  to  operational  requirements.  For  example,  the  controller  PECT 
was  to  be  developed  on  Windows  2000,  and  we  understood  that  it  is  problematic  to  achieve  the 
stated  tolerance  interval  on  that  platform  (given  that  Windows  2000  is  not  a  real-time  operat¬ 
ing  system).  Instead,  the  above  norms  were  selected  as  a  plot  device  for  our  study  of  empirical 
validation. 

8.  The  statistical  intervals  selected  for  these  model  problems  were  derived  from  Preiss  and  Wegmann’s  work  [Preiss  01]. 
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Although  the  model  problems  were  purposely  kept  quite  abstract,  two  technology  decisions 
were  made  in  the  interest  of  reducing  the  gap  between  the  model  solutions  and  their  ultimate 
deployment  environments.  The  operator  PECT  was  developed  for  the  ABB  Aspect  Integration 
Platform  (AIP).  The  AIP  is  a  development  and  deployment  environment  for  automation  sys¬ 
tems;  it  uses  OPC^  to  enable  operator  interfaces  to  communicate  with  controllers.  The  control¬ 
ler  PECT  was  developed  using  the  lEC  61850  standard  for  substation  automation. 

This  report  focuses  on  the  controller  PECT,  since  that  model  problem  posed  the  most  chal¬ 
lenges.  Although  the  latency  prediction  enabled  by  the  controller  PECT  is  quite  flexible  in 
defining  the  end  points  of  an  “end-to-end”  latency  prediction,  two  specific  controller  scenarios 
were  defined: 

1 .  switch  control  latency.  In  this  scenario,  we  are  interested  in  predicting  the  end-to-end 
latency  of  an  operation  to  manually  “throw”  the  SEI  switch. 

2.  over-current  protection  latency.  In  this  scenario,  we  are  interested  in  predicting  the  time 
required  to  detect  an  over-current  condition  and  automatically  “throw”  the  switch. 

These  scenarios  were  selected  in  part  because  they  correspond  to  actual  SAS  functionality,  as 
defined  in  lEC  61850. 


2.2  The  lEC  61850  Domain  Model 

The  lEC  61850  standard  defines  a  domain  model  of  the  functionality  of  substation  automation 
in  a  form  that  is  readily  adopted  for  use  with  software  component  technology  [I vers  02],  Fig¬ 
ure  2  depicts,  in  the  Unified  Modeling  Language  (UML),  the  EEC  61850  concepts  that  are 
most  important  for  SAS  model  problems.  An  SAS  consists  of  a  physical  system  and  a  logical 
system.  The  physical  system  comprises  primary  equipment,  such  as  switches,  breakers,  and 
transformers,  and  secondary  equipment,  such  as  intelligent  electronic  devices  (lEDs)  or  other 
controller  hardware.  The  logical  system  comprises  the  data  and  software  functions  for  Super¬ 
visory  Control  and  Data  Acquisition  (SCAD A),  Energy  Management  Systems  (EMSs),  and 
other  substation  applications. 

The  EEC  61850  standard  defines  a  variety  of  functions  and  the  quality  attributes  required  for 
each,  including  reliability,  accuracy,  and  latency.  Each  function  is  defined  in  terms  of  one  or 
more  logical  nodes  (LNs).  The  standard  defines  many  LNs;  each  one  is  a  primitive  building 
block  from  which  many  SAS  functions  may  be  composed.  Conceptually,  then,  there  is  a  close 
correspondence  between  LNs  and  software  components.  Although  it  is  possible,  and  perhaps 
reasonable,  to  deploy  several  LNs  as  a  single  component,  we  chose  to  maintain  a  one-to-one 
correspondence  between  LNs  and  the  software  components  that  implement  them. 


9.  OPC  stands  for  OLE  (Object  Linking  and  Embedding)  for  Process  Control. 


CMU/SE1-2002-TR-031  • 


7 


Figure  2:  Key  Concepts  of  the  I  EC  61850  Standard  for  a  SAS  Model  Problem 

The  lEC  61850  components  implemented  for  the  controller  PECT  are  summarized  in  Table  1. 
Only  rudimentary  functionality  was  implemented.  (Other  components  that  were  developed  but 
are  not  shown  here  include  an  OPC  gateway  for  communicating  with  the  operator  PECT  and  a 
clock  component  for  delivering  periodic  stimulus  to  controller  components.) 


Table  1:  lEC  61850  LNs  as  Components  of  the  Controller  PECT 


lEC  61850  LN 

Summary  of  Function 

Calculates  current 

TVTR 

Calculates  voltage 

MMXU 

Calculates  power 

CSWI 

Provides  safety  rules  for  the  controller  (e.g.,  select  before  use) 

XCBR 

Acts  as  the  controller  interface 

PIOC 

Provides  over-current  protection 
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2.3  Lab  Environment  and  the  SEI  Switch 

The  hardware  and  software  platforms  used  for  the  operator  and  controller  PECTs  were  con¬ 
ventional,  Intel-based  personal  computers  running  Microsoft’s  Windows  2000  operating  sys¬ 
tem.  The  operator  PECT  was  developed  for  Microsoft  .NET  in  C#.  The  controller  PECT  used 
VenturCom’s  RTX’°  real-time  extensions  package  to  support  priority-based  scheduling.  Other 
details  of  the  hardware  and  software  configuration  for  the  controller  PECT  are  described  in 
Table  13  and  Table  14  on  page  111. 

The  SEI  also  developed  a  low-power,  high-speed  switch  to  facilitate  the  prototype  develop¬ 
ment.  An  abstraction  of  this  switch  is  depicted  in  Figure  3;  the  actual  schematic  is  shown  in 
Figure  58  on  page  127. 


Figure  3:  SEI  Switch  Abstraction 

The  SEI  switch  communicates  with  the  software  controller  using  two  digital  input  channels  for 
commands  and  five  output  channels  (two  digital  and  three  analog)  for  status  and  values.  Digi¬ 
tal  input  is  provided  for  the  controller  to  select  the  switch  and  set  its  position  to  “open”  or 
“close.”  Analog  output  provides  positive  feedback  on  switch  selection  status  and  reports  on 
the  switch’s  current,  voltage,  and  position.  The  switch  is  implemented  as  a  serial  composition 
of  two  high-speed  electronic  switches,  which  are  also  known  as  TRIACs.  One  TRIAC  repre¬ 
sents  the  primary  equipment  being  controlled  (labeled  “Software  Switch”  in  Figure  3);  the 
other  is  controlled  internally  on  the  SEI  switch  and  used  for  over-current  protection  test  sce¬ 
narios  (labeled  “Hardware  Switch”  in  Figure  3).  The  hardware  switch  is  controlled  by  a  micro¬ 
processor  that  monitors  current  and  time.  If  the  current  exceeds  a  value  for  a  specified  duration 
(both  are  configurable),  the  microprocessor  “throws”  the  hardware  switch,  representing  a  fail¬ 
ure  of  the  PIOC  (see  Table  1)  component  to  protect  the  primary  equipment. 


1 0.  See  <http;//www.vcj.com>  for  details  about  this  software  package. 
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The  software  switch  is  operated  by  a  software  controller  running  on  the  Windows  2000  plat¬ 
form  through  a  data  acquisition  card  that  is  connected  to  the  analog  and  digital  lines.  This  con¬ 
troller  is  shown  in  detail  in  Figure  9  on  page  23. 


10 


CMU/SEI-2002-TR-031 


3  Prediction-Enabled  Component 
Technoiogy 


Our  approach  to  predictable  assembly  is  to  develop  and  use  prediction-enabled  component 
technologies  (PECTs).  We  formulated  the  basic  concepts  of  PECT  prior  to  undertaking  our 
SAS  effort  [Hissam  02],  but  refined  and  expanded  them  substantially  as  a  result  of  the  work 
reported  here. 

In  this  chapter,  we  first  outline  the  theory  of  PECT  in  Section  3.1  and  then  describe  the  struc¬ 
ture  of  a  PECT  implementation  in  Section  3.2.  Next,  we  provide  an  overview  of  Pin,  the  core 
component  model  we  use  to  build  PECTs,  in  Section  3.3.  Not  all  the  Pin  features  described  in 
this  report  were  implemented  in  the  model  solutions;  those  that  were  are  summarized  in  Sec¬ 
tion  3.4. 

In  the  following  discussion,  we  adopt  the  terminology  of  architecture  documentation  used  by 
Clements  and  associates  to  describe  PECT  [Clements  02b].  You  should  consult  their  work  if 
the  meanings  of  these  terms  are  not  self  evident.  Our  only  modification  to  this  terminology  is 
that,  in  some  situations,  we  prefer  the  term  assembly  to  view*  *  for  its  connotations  of  composi- 
tionality. 


3.1  The  Theory  of  PECT 

Component  composition,  in  the  literature,  almost  universally  assumes  an  underlying  compo- 
nent-and-connector  viewtype.**  Correspondingly,  a  PECT  has  a  central  component-and- 
connector**  viewtype,  called  the  constructive  viewtype  that  models  runtime  components  and 
their  interaction  topologies.  In  general,  we  denote  a  view  in  this  viewtype  as  a  constructive 
assembly.  The  constructive  viewtype  has  one  or  more  analysis  viewtypes,  called  analysis 
views,  associated  with  it.  Each  analysis  viewtype  provides  a  basis  for  compositional  reason¬ 
ing  about  some  runtime  behavior,  for  example,  latency,  security,  safety,  liveness,  or  reliability. 


11.  As  defined  by  Clements  and  associates  [Clements  02b]. 

1 2.  In  previous  documents,  we  referred  to  analysis  views  as  analysis  assemblies.  View  seems  preferable  to  assembly  here,  be¬ 
cause  assembly  has  a  stronger  connotation  of  components  and  connectors  than  can  be  justified  In  the  general  case  of  views 
that  support  behavioraj  analysis  technology.  However,  in  this  case  study,  the  term  analysis  assembly  \urns  out  to  be  appro¬ 
priate. 
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In  PECT,  analysis  views  are  derived  automatically  from  constructive  assemblies  by  means  of  a 
syntactic  transformation,  called  an  interpretation  (see  Figure  4).  An  interpretation  is  a  partial 
function  from  constructive  assemblies  to  analysis  views.  Three  interpretations  are  shown  in 
Figure  4:  /latency*  ^SAFETY*  ^avail- 


A^ATENCY  j 

Latency 

1 

Analysis  View 

Constructive 

Safety 

Safety  Analysis 

Assembly 

w 

1 

View 

^AVAIL 

Availability 

1  ! 

] 

Analysis  View 

Figure  4:  Interpretations  on  Constructive  Assemblies  and  Analysis  Views 

Interpretations  may  impose  constraints  on  constructive  assemblies  beyond  those  specified  in 
the  constructive  viewtype,  for  example,  restrictions  on  allowed  topologies  of  component  inter¬ 
actions  or  on  component  behavior.  Interpretations  may  also  require  analysis-specific  compo¬ 
nent  information  that  is  not  defined  in  the  constructive  viewtype.  For  example,  a  latency 
analysis  viewtype  might  require  the  priority  assignment  of  component  execution  threads. 

An  interpretation  is  consistent  if  each  constructive  assembly  is  related  to,  at  most,  one  analysis 
view  under  that  interpretation.  An  interpretation  is  valid  if  there  is  empirical  or  formal  justifi¬ 
cation  for  the  claim  that  behaviors  predicted  in  the  analysis  view  will  be  manifested  by  the 
assembly  of  components  specified  in  the  constructive  view.  A  PECT,  then,  consists  (in  part)  of 
a  set  of  consistent  and  valid  interpretations. 


3.2  The  Structure  of  PECT 

Figure  5  depicts  the  four  different  environments  in  PECT  and  their  conceptual  dependencies  in 
terms  of  a  UML  class  diagram.  As  implied  by  its  name,  a  PECT  consists  of  a  component  tech¬ 
nology  that  has  been  extended  with  one  or  more  prediction-enabling  technologies.  We  do  not 
claim  to  have  discovered  a  universally  acceptable  definition  of  these  technologies;  instead,  we 
define  them  relative  to  PECT.  The  assembly  and  runtime  environments  are  provided  by  a  com¬ 
ponent  technology  discussed  in  Section  3.2.1,  while  the  analysis  and  validation  environments 
are  provided  by  a  prediction-enabling  technology  discussed  in  Section  3.2.2. 
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Component 

Technology 


Prediction 

Enablers 


Figure  5:  The  Four  Environments  of  a  PECT 

3.2.1  Component  Technology 

A  component  technology  comprises  an  assembly  environment  and  a  runtime  environment.  In 
current  literature  and  commercial  products,  a  component  runtime  environment  is  often 
referred  to  as  a  container.  We  prefer  our  own  terminology  here. 

•  The  assembly  environment  supports  development-time  activities  such  as  selecting  compo¬ 
nents,  configuring  component  properties,  and  connecting  components  into  assemblies.  An 
assembly  environment  supports  pure  composition  if  those  operations  are  the  only  ones 
required  to  develop  applications.  Pure  composition  involves  developing  applications 
entirely  from  components  and  connectors. 

•  The  runtime  environment  manages  the  execution  of  components  (e.g.,  their  execution 
schedules  and  life  cycles),  manages  resources  shared  by  components,  and  provides  ser¬ 
vices  that  allow  components  to  interact  with  the  external  world. 

Assemblies  are  deployed  in  their  runtime  environment,  that  is,  by  a  mechanism  either  native  or 
external  to  the  component  technology. 

Not  shown  in  Figure  5  is  the  fact  that  both  of  the  above  environments  are  specialized  to  sup¬ 
port  a  particular  constructive  model.^^  The  constructive  model  specifies  the  types  of  compo¬ 
nents  in  an  assembly,  their  type-specific  runtime  behavior,  and  the  constraints  on  their 
connection  topology  [Bachmann  00].  In  our  view,  this  concept  of  constructive  model  is  nearly 
(if  not  exactly)  identical  to  that  of  architectural  style,  as  that  term  has  been  used  in  current  lit¬ 
erature  [Gardiner  00],  [Bass  98],  [Clements  02b].  Therefore,  the  Pin  constructive  model,  or. 


13.  Our  use  of  the  term  constructive  model  supersedes  the  earlier,  and  more  ambiguous,  term  component  model. 
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more  ponderously,  the  Pin  constructive  component  model,  is  also  sometimes  referred  to  as  the 
Pin  style.  Pin  is  described  in  Section  3.3. 


3.2.2  Prediction  Enabiers 

Prediction  enablers  comprise  an  analysis  environment  and  a  validation  environment: 

•  The  analysis  environment  supports  reasoning  about  the  runtime  behavior  of  a  system. 
Simplistically,  reasoning  is  compositional  if  it  supports  “divide-and  conquer”  analysis. 

•  The  validation  environment  is  an  infrastructure  for  controlled,  experimental  validation  of 
predictions  of  assembly  behavior  made  in  the  analysis  environment. 

The  association  between  the  analysis  and  validation  environments  shown  in  Figure  5  reflects  a 
logical  dependency  between  the  theoretical  model  underlying  a  particular  form  of  design  anal¬ 
ysis  and  the  experimental  apparatus  needed  to  experimentally  validate  the  theory. 


3.2.3  PECTsComponent  Technology+Analysis  Technology 

We  enable  reasoning  in  a  component  technology  by  extending  its  assembly  environment  with 
one  or  more  analysis  environments.  This  extension  is  implemented  via  a  plug-in  interface.  An 
analysis  environment  provides  an  implementation  that  satisfies  the  plug-in,  which,  in  turn, 
allows  users  of  an  assembly  environment  to  reason  about  as  many  assembly  properties  as  there 
are  plug-ins. 

We  enable  the  validation  of  analysis  technology  by  extending  the  component  runtime  with  one 
or  more  validation  environments.  This  extension  is  also  implemented  by  means  of  a  plug-in 
interface,  although  in  this  case,  we  do  not  envisage  multiple  validation  environments  operating 
concurrently  (but  you  never  know).  The  observation  mechanism  is  based  on  execution  traces 
where  observable  trace  events  (e.g.,  procedure  calls  or  message  exchanges)  have  been  anno¬ 
tated  with  property-theory-specific  information. 


3.3  The  Pin  Style 

Pin  allows  us  to  emphasize  the  compositional  potential  of  software  component  technology. 
This  may  sound  tautological  at  first,  but,  with  a  little  study,  it  becomes  apparent  that  compo¬ 
nents  and  composition  do  not  always  go  hand  in  hand.  Consider,  for  example,  the  amount  of 
“glue”  code  that  must  often  be  developed,  in  conventional  programming  languages,  to  inte¬ 
grate  components  [Wallnau  02].*^  With  Pin,  we  can  explore  the  potential  for  pure  composi¬ 
tion.  It  is  our  view  that  breakthroughs  in  programmer  productivity  and  system  quality  require 

14.  By  “glue"  code,  we  mean  something  messy  and  ad  hoc  that  integrators  cobble  together  as  needed.  While  connectors  are 
also  a  form  of  “glue,"  we  typically  think  of  connectors  as  being  well  defined  and  provided  by  a  compiler  or  Infrastructure. 
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higher  level  programming  abstractions,  and  that  pure  composition  has  many  of  the  qualities 
we  might  want  such  abstractions  to  possess,  especially  when  it  is  “prediction  enabled.” 

The  Pin  style  (hereafter  referred  to  as  simply  Pin)  adopts  the  hardware  metaphor  that  compo¬ 
nents  interact  with  their  environment  through  a  set  of  pins.  Components  in  Pin  are  specified  as 
a  set  of  pins  and  pin  behaviors;  assemblies  are  specified  as  a  set  of  connections  among  pins. 
Pin  defines  a  number  of  connectors,  each  specialized  to  a  particular  type  of  pin  and  pin-to-pin 
interaction  scheme.  In  this  report,  we  provide  a  high-level  overview  of  the  graphical  notation 
for  Pin  and  the  connectors  used  in  SAS  PECTs.  A  more  formal  and  detailed  treatment  is  pro¬ 
vided  by  Ivers  and  associates  [I vers  02]. 

We  describe  the  notation  of  components  and  pins  in  Section  3.3.1,  assemblies  and  environ¬ 
ments  in  Section  3.3.2,  the  behavioral  specification  of  components  and  the  composition  of 
component  behavior  in  Section  3.3.3,  and  assemblies  of  assemblies  in  Section  3.3.4. 


3.3.1  Components  and  Pins 

A  component  interacts  with  the  external  world  exclusively  through  its  pins-,  there  are  no  other 
communication  paths  to  or  from  a  component.  There  are  two  types  of  pins:  sink  pins  and 
source  pins.  A  component  receives  communication  (stimuli)  from  the  environment  via  its  sink 
pins  and  sends  communication  (responses)  to  the  environment  via  its  source  pins.  Figure  6 
depicts  the  graphic  notation  we  use.  As  suggested  by  the  different  shapes  given  to  pin  heads  in 
Figure  6,  there  are  different  types  of  source  and  sink  pins.  These  types  and  their  notations  are 
defined  in  the  following  sections. 


Figure  6:  Pin  Notation 

Components  are  denoted  by  Cj,  sink  pins  are  denoted  by  Sj  (for  stimulusj),  and  source  pins  are 
denoted  by  rj  (for  responsej).  Where  the  distinction  between  sink  and  source  pin  is  irrelevant, 
we  use  Pj  (for  pinj).  In  all  cases,  we  omit  subscripts  where  they  are  not  required.  Component 
names,  which  must  be  unique,  are  also  used  as  pin  names.  The  expression  c.p  denotes  pin  p  of 
component  c.  We  use  capitalized  names  to  denote  the  predicates  defined  on  pins.  For  example. 
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SomeCondition(c.p)  is  true  if  pin  c.p  satisfies  SomeCondition,  and  is  false  otherwise.  We  use 
lowercase  names  to  denote  properties  of  pins.  For  example,  someProperty(c.p)  denotes  the 
value  of  pin  c.p’s  someProperty  property. 


3.3.1. 1  Pin  Data  Interface 

All  pins  have  a  data  interface,  denoted  by  interface(c.p).  This  corresponds  to  what  is  usually 
called  the  signature  of  a  method  and  defined  as  a  sequence  of  formal  arguments  <name,  type, 
mode>,  where  mode  is  one  of  {In,  Out,  InOut},  and  type  is  one  of  {Boolean,  SByte,  UByte, 
SWord,  UWord,  SDWord,  UDWord,  SDouble,  Float,  String}.  For  simplicity,  we  do  not  define 
the  representation  of  these  types  in  this  report. 

3.3.1 .2  Source  Pins 

There  are  two  types  of  source  pins:  asynchronous  and  synchronous.  Informally,  an  asynchro¬ 
nous  source  pin  represents  the  ability  of  a  component  to  make  a  request,  where  the  component 
does  not  wait  (or  block)  for  the  request  to  be  satisfied.  A  synchronous  source  pin  also  repre¬ 
sents  the  ability  to  make  a  request  by  a  component,  but  in  this  case,  the  component  must  wait 
for  the  request  to  be  satisfied.  Asynchronous  source  pins  are  graphically  denoted  with  the  > 
pin  head,  and  synchronous  ones  use  the  >  pin  head.  In  Figure  6,  c.rj  is  an  asynchronous  source 
pin,  while  c.r2  is  a  synchronous  source  pin.  It  is  reasonable  to  think  of  asynchronous  source 
pins  as  message  sends  and  synchronous  source  pins  as  procedure  calls,  but  you  shouldn’t  put 
too  much  faith  in  this  gross  interpretation;  in  fact,  that  is  not  how  these  pins  were  implemented 
in  the  SAS  model  solutions.  Nonetheless,  arguments  in  an  asynchronous  pin  data  interface 
must  be  of  mode  In. 

If  Mandatory(c.r),  then  c.r  is  a  mandatory  source  pin;  otherwise  it  is  optional.  Optional  source 
pins  need  not  be  connected  in  an  assembly,  while  mandatory  source  pins  must  be  connected. 
This  models  the  distinction  between  uses  (mandatory)  and  calls  (optional),  respectively.  A 
component  Cq  uses  sink  pin  Cj.s  if  the  behavior  of  Cq  depends  on  the  correct  behavior  of  Cj.s; 
otherwise  it  only  calls  Cj.s.  Graphically,  we  distinguish  optional  source  pins  from  mandatory 
ones  by  enclosing  the  names  of  optional  sources  in  square  brackets  ([  ]).  So,  c.[r2]  is  an 
optional  source  pin  in  Figure  6. 


3.3.1 .3  Sink  Pins 

There  are  two  types  of  sink  pins:  asynchronous  and  synchronous,  referring  to  a  component’s 
ability  to  receive  asynchronous  or  synchronous  communication,  respectively.  As  with  source 
pins,  the  data  interface  of  an  asynchronous  sink  pin  may  include  only  arguments  of  mode  In. 
Asynchronous  and  synchronous  sink  pins  are  graphically  denoted  with  pin  heads  >  and  >, 
respectively.  In  Figure  6,  c.Sj  is  asynchronous,  while  c.sj^,  2  <  k  <  6,  are  synchronous. 
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If  Threaded(c.s),  then  c.s  has  its  own  thread  of  control,  and  threadld(c.s)  denotes  its  identity. 
Threads  represent  units  of  concurrent  execution  and  may  be  implemented  by  operating  system 
threads,  processes,  tasks,  and  so  forth.  Graphically,  tj  =  threadld(c.s)  is  denoted  as  a  suffix  :  tj 
on  the  sink  pin  name.  In  Figure  6,  sink  pins  c.S2  and  C.S3  are  unthreaded.  Threads  may  be 
shared  by  sink  pins.  So,  in  Figure  6,  C.S5  and  c.Sg  share  thread  t3,  but  c.Sj  and  C.S4  have  their 
own  threads.  Note  that  asynchronous  sink  pins  must  be  threaded,  that  is,  Asynchronous(c.s) 
=>Threaded(c.s). 

If  Mutex(c.s),  then  c.s  is  called  a  mutex  sink,  and  only  one  caller  may  be  active  on  c.s  at  any 
given  time — in  effect,  the  caller  must  obtain  the  semaphore  for  c.s.  Conversely,  if  — iMu- 
tex(c.s),  then  c.s  is  called  a  reentrant  sink  and  is  never  guarded  by  a  semaphore.  Note  that  even 
a  reentrant  c.s  might  force  a  caller  to  wait  while  c.s  synchronizes  on  an  internal  (to  c.s) 
resource.  This  characteristic  pertains  only  to  synchronous  sink  pins.  Mutex  sinks  are  repre¬ 
sented  by  >|.  In  Figure  6,  Mutex(c.S]^),  where  3  <  k  <  6.  Note  that  Threaded(c.s)  a  — lAsyn- 
chronous(c.s)  =>  Mutex(c.s). 

3.3.2  Assemblies  and  Environments 

An  assembly  is  defined  as  a  set  of  components  and  their  connections.  We  denote  an  assembly 
as  aj,  and  a.c  denotes  the  component  c  in  the  scope  of  assembly  a.  An  assembly  is  graphically 
depicted  as  a  box  enclosing  a  set  of  connected  components,  which  visually  reinforces  the 
notion  that  assemblies  are  composed  of  components.  Figure  7  depicts  a  simple  assembly  to 
illustrate  key  points  in  the  following  discussion. 

A  connection,  or  connector,  is  established  between  two  components  when  some  source  pin  Cj.r 
is  connected  to  some  sink  pin  Cj.s,  i  9^  j  (this  inequality  is  assumed  in  further  discussion).  Com¬ 
ponents  Cj  and  Cj  must  be  part  of  the  same  assembly  if  they  are  to  be  composed.  In  text,  we 
denote  the  connection  as  a(Cj.r  Cj.s),  although  we  usually  omit  the  a()  where  doing  so  will 
not  cause  confusion.  A  connection  Cj.r  Cj.s  requires  that  Cj.r  and  Cj.s  be  mutually  conform¬ 
ant.  The  rule  for  mutual  conformance  is  simple:  both  pins  must  be  synchronous,  or  both  must 
be  asynchronous,  and  their  data  interfaces  must  have  the  same  argument  types,  modes,  and 
positions.  Graphically,  we  denote  a  connection  as  a  double-headed  lollipop,  with  the  lollipop 
heads  circumscribing  the  connected  pins.  In  Figure  7,  CQ.r  C2.s  is  a  connection. 
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assembly  name  environment  type 


Figure  7:  A  Simple  Assembly 

Each  assembly  has  an  associated  environment  type,  which  is  denoted  by  the  :Ej  suffix  of  an 
assembly  name.  Environment  types  play  two  crucial  roles  in  Pin.  First,  they  define  the  services 
(specified  as  sink  and  source  pins)  that  components  may  use.  Graphically,  these  services  are 
attached  to  the  assembly  by  means  of  environment  junctions,  drawn  as  small  black  boxes.  For 
example,  in  Figure  7,  a  runtime  environment  associated  with  assembly  oq  provides  two  ser¬ 
vices:  the  source  pin  CLOCK  and  the  sink  pin  CONSOLE.  The  environment’s  services  can  be 
thought  of  as  being  implemented  by  an  environment-provided  component  with  a  single  sink 
pin  (aQ.CONSOLE)  and  source  pin  (aQ.CLOCK).  We  denote  a  connection  with  an  environment 
service  in  an  analogous  way  to  that  defined  earlier,  that  is,  as  ao-CLOCK  ao.Co.si  and 
ao.C3.r]  aQ.CONSOLE.  To  make  these  expressions  easier  to  read,  we  allow  assembly  names 
to  distribute  over  so,  instead  of  the  above,  we  could  write  ao(CLOCK  Cq.Sj)  and 
ao(c3.rj  CONSOLE).  Again,  we  might  omit  aQ  if  doing  so  does  not  cause  confusion. 

Second,  environment  types  supply  the  interaction  models  implemented  by  connectors.  For 
example,  consider  the  interaction  topology  in  Figure  7  that  includes 

•  CLOCK  Cq.Sj,  C3.rj  CONSOLE,  C2.r2  C0.S2 

•  CQ.r-^  {c,.s,C2.s} 

•  {cj.r,  C2.ri}  ^  C3.S 

where  we  use  sets  {c.p],  c.p2,...}  to  denote  multiple  sinks  and  sources  on  1:N  and  N:1  compo¬ 
sitions,  respectively.  The  semantics  of  1:1  composition  (the  first  bulleted  item,  above)  is  likely 
to  be  clear  intuitively,  but  what  about  the  1:N  asynchronous  composition  (the  second  bulleted 
item)?  How  many  connectors  are  there?  Is  a  broadcast  or  a  sequence  of  unicasts  represented? 
If  a  sequence  is,  what  is  its  order?  What  is  the  semantics  of  message  buffering — first  in,  first 
out  (FIFO)  or  last  in,  last  out  (LIFO)?  What  is  the  capacity  of  the  message  buffers?  Analo- 
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gously,  what  is  the  interaction  order  of  the  N:1  synchronous  composition?  We  may  want  dif¬ 
ferent  answers  to  these  questions  in  different  component  runtime  environments.  Ivers  and 
associates  give  a  detailed  treatment  of  how  these  semantics  are  defined  [Ivers  02]. 


3.3.3  Behavior:  Reaction  and  Interaction 

In  PECT,  the  semantics  of  composition  is  defined  by  analysis  views.  That  is,  we  define  the 
semantics  of  an  assembly  as  the  observable  behavior  of  an  assembly.  Informally,  the  Pin  syn¬ 
tax  describes  what  an  assembly  looks  like,  while  the  semantics  describes  what  it  does  at  runt¬ 
ime.  We  say  that  Pin  is  semantically  extensible  since  different  syntactic  elements  of  Pin  may 
be  annotated*^  with  information  that  is  used  to  construct,  via  interpretations,  the  semantics  of 
each  assembly  in  the  Pin  style. 

Nonetheless,  one  semantics  is  of  such  utility  to  predictable  assembly  that  it  is  built  into  Pin — 
the  behavior  that  is  specifiable  in  a  process  algebra  such  as  Hoare’s  CSP  [Hoare  85],  Magee 
and  Kramer’s  ESP  [Magee  99],  or  Milner’s  CCS  [Milner  89]  and  7t-Calculus  [Milner  99].  We 
have  chosen,  at  this  time,  to  use  CSP  as  the  behavior  specification  language  for  Pin.  In  this 
report,  we  describe  only  the  essence  of  our  approach.  For  complete  details  on  it,  see  A  Basis 
for  Composition  Language  CL  [Ivers  02].  We  first  treat  the  specification  of  component  behav¬ 
ior  in  the  form  of  reactions  and  then  treat  the  composition  behavior  in  the  form  of  interactions. 

3.3.3.1  Component  Behavior:  Reactions 

Components  have  intrinsic  behavior,  as  defined  by  their  implementations.  We  model  the 
behavior  of  a  component  in  terms  of  reactions.  A  reaction  is  a  CSP  process  that  relates  one  or 
more  sink  pins  to  one  or  more  source  pins,  indicating  how  the  component  reacts  to  the  stimula¬ 
tion  of  its  sink  pins.  The  general  form  is  a  process  that  looks  something  like  R  =  s  — >  r  — >  R, 
where  s  is  a  sink  pin  that  can  be  stimulated,  r  is  a  source  pin  that  is  stimulated  in  response  to  an 
interaction  on  s,  and  R  is  the  CSP  process  that  describes  this  pattern  of  behavior.  Note  that 
reactions  are  defined  within  the  scope  of  a  component,  so  the  usual  denotation  of  c.s,  c.r,  c.R 
and  so  on  is  superfluous. 

Reactions  also  reflect  the  thread  structure  of  a  component.  For  example,  all  behavior  imple¬ 
mented  by  a  coimnon  thread  is  modeled  as  a  single  reaction.  This  allows  analyses  to  take  into 
account  the  actual  degree  of  threading,  and  potential  concurrency  errors,  of  the  component 
implementation.  A  component’s  behavior  as  a  whole  is  specified  as  the  CSP  parallel  (indicated 
by  the  ||  symbol)  or  interleaved  (indicated  by  the  |||  symbol)  composition  of  its  reactions, 
depending  on  which  better  models  the  actual  interaction  among  the  component’s  threads. 


1 5.  The  annotation  mechanism  or  syntax  is  not  described  in  this  report. 
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The  gist  of  reaction  rules  is  depicted  in  Figure  8. 
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Figure  8:  Reactions  Specify  Component  Behavior 

In  this  example,  there  are  three  reactions:  Rj,  R2,  and  R3.  The  ovals  are  used  to  illustrate 
which  pins  are  related  by  each  reaction;  for  example,  R,  is  shown  as  relating  sink  pins  Sj  and 
$2  to  source  pins  rj  and  r2.  Rj  represents  the  behavior  of  a  single  thread  t  that  is  used  to  process 
sinks  Sj  and  S2.  A  definition  of  R|  could  be  Rj  =  (sj  — »  rj  — >  Rj)  □  (S2  — >  r2  — ^  Rj)  where  □ 
means  external  choice. 

Reactions  allow  us  to  specify  the  causal  dependencies  among  behaviors  in  an  assembly  of 
components.  The  most  basic  causal  dependency  is  the  dependency  chain,  as  illustrated  in  the 
above  reaction.  In  fact,  this  is  the  only  causal  dependency  we  required  for  the  latency  analysis 
in  SAS  model  solutions.  More  complex  behaviors,  such  as  coordination  among  reactions  or 
changes  in  behavior  based  on  accumulated  state  information,  can  also  be  modeled. 

3.3.3.2  Assembly  Behavior:  Interactions 

Up  to  this  point,  we  have  been  using  the  operator  in  an  informal  manner.  Although  we 
never  made  the  claim,  the  reader  might  infer  that  has  some  sort  of  algebraic  properties  and 
associated  composition  semantics.  A  simple  algebraic  model  for  might  be 
<R,  •  a"  G  E}) ,  where  R  denotes  reactions,  and  in  place  of  the  single  operator  we  have  a 

set  of  operators  ,  one  for  each  (n**’)  interaction  scheme  a"  defined  for  an  environment  E, 

where  the  operators  of  a"  may  be  of  arbitrary  arity. 

So,  for  example,  from  Figure  7,  the  1:N  interaction  aoCcg.r  -^{cj.s,  C2.s})  denotes  the  syntac¬ 
tic  composition  of  three  components  cq,  Cj,  and  C2,  in  assembly  aQ,  on  pins  Co.r,  Cj.s,  and  C2.S. 
The  interaction  scheme  for  this  composition  is  >,  as  defined  in  environment  type  Eq.  We  can¬ 
not  tell  whether  is  a  binary  (unicast)  or  n-ary  (broadcast)  operator.  This  and  other  aspects  of 
the  semantics  of  this  interaction  must  be  formally  specified. 

Such  specifications  are  not  trivial,  but  there  are  specification  patterns  that  can  simplify  them. 
One  objective  of  A  Basis  for  Composition  Language  CL  [Ivers  02]  is  to  produce  a  general 
approach  for  specifying  the  composition  semantics  for  arbitrary  interaction  schemes  (connec¬ 
tors)  for  arbitrary  environment  type  E.  We  hasten  to  add  here  that  end  users  of  PECTs  never 
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see  these  semantic  complexities,  any  more  than  users  of  modem  programming  languages  see 
the  complexity  of  type  checking  or  code  generators. 

3.3.4  Hierarchical  Assembly 

So  far  we  have  described  a  component  model  that  is  quite  flat:  components  can  interact  only 
with  components  in  the  same  assembly  and  only  with  a  single  runtime  environment  associated 
with  that  assembly.  Pin  will  not  scale  to  interesting  problems  without  introducing  some  form 
of  hierarchical  composition.  In  fact,  the  algebraic  model  (R,  •  a"  e  E})  has  a  hierarchical 

meaning,  because  R3  =  Rj-^R2  (for  some  arbitrary  binary  composition  operator  and  reac¬ 
tions  R{  1,2,3})  induces  a  hierarchy  that  is  rooted  at  R3  and  composes  Ri  and  R2. 

Hierarchical  assembly  in  Pin  always  involves  treating  an  assembly  as  a  component.  That  is, 
the  assembly  has  an  interface  defined  in  terms  of  source  and  sink  pins.  The  correspondence 
between  component  pins  and  assembly  pins  is  established  by  means  of  assembly  junctions. 
Two  forms  of  junctions  are  deHned:  null  junctions  and  gateway  junctions.  We  note  at  the  out¬ 
set  that  hierarchical  composition  introduces  many  subtle  complexities  not  generally  addressed 
by  component  technology.  We  do  not  discuss  these  subtle  issues  in  this  report  because  our 
focus  is  on  the  controller  assembly  only. 


3.4  Pin  Subset  in  SAS  Model  Solutions 

The  preceding  description  of  Pin  is  more  general  and  complete  than  the  version  of  Pin  used  in 

the  SAS  PECTs — many  of  the  generalities  were  a  result  of  our  experience.  The  following  sum¬ 
marizes  those  aspects  of  Pin  that  were  actually  used: 

•  The  operator  PECT  used  Microsoft  .NET  as  its  component  runtime,  while  the  controller 
PECT  used  Microsoft  Windows  2000  augmented  with  support  for  real-time,  priority- 
based  scheduling.  Differences  between  these  runtimes  were  reflected  in  the  latency  theo¬ 
ries  used,  but  not  in  composition  semantics. 

•  The  assembly  environments  for  the  operator  and  controller  PECTs  were  conventional  text 
editors  operating  on  component  and  assembly  descriptions  written  in  Extensible  Markup 
Language  (XML). 

•  The  operator  and  controller  PECTs  had  an  analysis  environment  in  the  limited  sense  that 
latency  prediction  was  automated,  and  both  had  validation  environments  based  on  traces 
of  time-stamped  pin  activations. 

•  All  the  pin  types  described  in  Section  3.3.1  were  implemented.  However,  components 
were  restricted  to  one  thread  of  control  that  was  shared  by  all  asynchronous  sink  pins.  Fur¬ 
ther,  synchronous  sink  pins  were  not  threaded. 
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•  Sink  pins  were  annotated  (in  XML)  with  execution  time.  Components  were  annotated 
with  a  single  execution  priority.  Callers  on  synchronous,  mutexed  sink  pins  would  acquire 
the  priority  level  of  the  called  component. 

•  The  notion  of  assemblies  was  implicit — although  assemblies  were  units  of  deployment, 
they  were  not  named  entities.  Environment  types  did  not  appear  at  all.  Environment  ser¬ 
vices,  such  as  periodic  timers,  were  implemented  as  environment-provided  components. 

•  OPC  gateway  junctions  were  used  to  (hierarchically)  compose  an  assembly  of  the  operator 
PECT  with  an  assembly  of  the  controller  PECT;  analysis  hierarchies  using  null  junctions 
were  not  used. 

•  The  CSP  reactions  were  specified,  on  paper,  for  each  controller  component.  These  reac¬ 
tions  were  then  used  to  manually  construct  the  models  used  in  the  temporal-logic  model 
checking,  as  described  in  Chapter  7  and  Appendix  C,  and  the  ordering  of  interactions  over 
1:N  connection  topologies  for  controller  latency  prediction. 

•  There  were  no  formal  compositional  semantics  defined  for  the  connectors. 

3.5  SAS  Software  Controller  in  Pin 

Figure  9  depicts  the  SAS  software  controller  developed  in  Pin  for  this  model  problem. 

The  names  and  functional  roles  played  by  the  components  in  Figure  9  (i.e.,  CSWI,  TCTR, 
TVTR,  MMXU,  XCBR,  and  PIOC)  are  defined  by  the  lEC  61850  standard.  SwMonSource 
and  SwMonSink  manage  the  interface  to  the  hardware  switch.  Clock  is  a  component  that  is 
(logically)  bundled  with  the  latency  model,  and  OPCGateway  allows  us  to  create  an 
assembly  of  assemblies — one  that  allows  the  operator  and  controller  assemblies  to  interact 
using  the  OPC  protocol. 
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1 6.  Each  component  In  this  figure  had  at  most  one  thread  of  execution  that  was  shared  among  ail  asynchronous  sink  pins.  Those 
threads  are  not  depicted  in  this  figure  for  the  sake  of  simplicity. 
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4  PECT  Process  and  Workflows 


4.1  The  PECT  Development  Process 

We  develop  and  validate  a  PECT  using  a  process  that  consists  of  four  fundamental  phases: 

1 .  Definition 

2.  Co-Refinement 

3.  Validation 

4.  Packaging 

During  the  Definition  phase,  we  define  the  functional  requirements,  the  assembly  property  to 
analyze,  and  the  statistical  goals  for  the  PECT.  Once  those  goals  have  been  set,  we  create  a 
component  model,  an  analysis  model,  and  a  measurement  framework  through  an  iterative  pro¬ 
cess  known  as  co-refinement.  A  PECT  instance,  which  integrates  the  component  and  analysis 
models,  emerges  from  co-refinement.  During  the  Validation  phase,  we  validate  the  PECT 
instance  against  the  defined  goals.  If  it  meets  the  goals  sufficiently,  the  PECT  is  packaged  and 
released.  If  the  goals  are  not  met,  co-refinement  is  repeated  until  they  are.  Figure  10  shows  a 
brief  overview  of  the  PECT  process.  As  the  arrows  indicate,  this  process  is  highly  iterative. 
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Figure  10:  An  Overview  of  the  PECT  Development  Method 

In  the  following  sections,  we  translate  the  PECT  process  concretely  into  (using  Kruchten’s 
terms)  deliverables,  workers,  activities,  artifacts,  and  workflows  [Kruchten  00].  Deliverables 
constitute  the  primary  output  of  the  PECT  process.  Workers  are  those  involved  in  performing 
the  PECT  process.  Artifacts  describe  what  is  produced  during  the  PECT  process.  Activities 
describe  how  the  PECT  process  is  accomplished,  and  workflows  provide  insight  into  when  in 
the  process  they  occur. 


4.1.1  Deliverables 

The  outcome  of  the  process  is  a  PECT  that  consists  of 

•  a  component  technology 

•  one  or  more  analysis  views  and  their  associated  interpretations 

•  a  validation  environment  or  other  means  of  establishing  trust  in  analysis  and  predictions 

•  validation  data  and  statistical  labels 
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The  delivered  PECT  has  the  following  characteristics: 


•  zero  programming  assembly 

Components  can  be  assembled  into  applications  without  additional  programming.  If  addi¬ 
tional  code  has  to  be  provided,  the  code  must  be  encapsulated  in  PECT-compliant  compo¬ 
nents. 

•  automatic  interpretation 

To  perform  predictions  in  an  execution  environment  where  the  end  assemblies  are 
unknown,  the  PECT  must  be  able  to  interpret  a  constructive  assembly  into  an  analysis 
view  automatically.  The  analysis  view  is  then  used  to  perform  the  desired  prediction. 

•  objective  trust 

Objective  trust  is  achieved  by  validating  the  prediction  theory  using  a  wide  variety  of 
assemblies.  Statistical  means  can  be  used  to  calculate  how  many  assemblies  have  to  be 
executed  to  meet  the  statistical  goals  set  for  the  prediction  theory. 


4.1 .2  Workers,  Activities,  and  Artifacts 

In  the  Rational  Unified  Process  (RUP),  an  activity  is  generally  assigned  to  a  specific  worker 
and  has  a  duration  of  a  few  hours  to  a  few  days  [Kruchten  00];  we  relax  this  stipulation  some¬ 
what  and  allow  multiple  workers  to  contribute  to  an  activity,  where  that  activity  has  an  indeter¬ 
minate  duration.  As  with  the  RUP,  each  activity  has  a  clear  purpose,  usually  centered  on 
creating  or  modifying  artifacts — the  tangible  products  of  the  PECT  development  method. 

Table  2  depicts  the  major  artifacts  of  the  PECT  development  process  that  are  the  result  of 
workers  performing  activities.  The  fact  that  each  artifact  is  assigned  to  just  one  worker  should 
be  interpreted  not  as  an  exclusive  assignment,  but  rather  as  an  assignment  of  primary  responsi¬ 
bility.  In  general,  multiple  workers  are  required  for  most  artifacts.  For  example,  defining  the 
requirements  for  a  PECT  is  primarily  the  responsibility  of  the  customer,  but  setting  the  appro¬ 
priate  requirements  requires  the  contributions  of  the  PECT  designer,  attribute  specialist,  and 
component  model  specialist. 


Table  2:  PECT  Development  Activities 


Activity 

Workers 

Artifacts 

Define  functional  requirements 

customer 

PECT  requirements 

Define  assembly  property 

customer 

assembly  property  definition 

Define  statistical  goals 

customer 

normative  confidence  labels 

Define  component  model 

component  model  specialist 

constructive  viewtype 

Define  analysis  model 

attribute  specialist 

analysis  viewtype 

Define  property  theory 

attribute  specialist 

property  theory  and  its  valida¬ 
tion  concept 
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Table  2:  PECT  Development  Activities  (Cont’d.) 


Activity 

Workers 

Artifacts 

Validate  theory  empirically 

measurement  specialist 

experiment  design,  statistical 
labels 

Develop  analysis  interpretations 

PECT  designer 

component  and  assembly 
description  schemas;  interpreta¬ 
tions 

Develop  component  measure¬ 
ment  apparatus 

component  developer 

measurement  specialist 

test  components  (synthetic, 
actual) 

testbench  for  component  and 
assembly  measurement 

Develop  assembly  measurement 
apparatus 

measurement  specialist 

assembly  generator  and  analysis 
tools 

Create  validation  sample 

component  developer 

application  assembler 

any  additional  assembly  compo¬ 
nents 

sample  assembly  set 

Obtain  component 
measurements 

measurement  specialist 

component  measurements 
(labels) 

Obtain  assembly  measurements 

measurement  specialist 

assembly  measurements  and 
analysis  (labels) 

Deploy  PECT 

PECT  designer 

packaged  PECT 

Below,  we  describe  workers  whose  roles  are  not  implicitly  clear  by  their  names  alone: 


•  component  model  specialist:  defines  the  constructive  viewtype  used  in  the  PECT.  This 
person  may  or  may  not  be  the  designer  of  a  proprietary  component  technology.  Either  way, 
if  the  component  technology  has  already  been  defined,  this  specialist  will  have  in-depth 
knowledge  about  it. 

•  attribute  specialist:  defines  and  helps  to  validate  both  the  property  theory  and  the  analysis 
model.  This  person  must  know  the  theories  behind  the  chosen  attribute  (emergent  prop¬ 
erty)  that  the  PECT  is  engineered  to  predict.  For  example,  if  the  attribute  is  latency,  the 
attribute  specialist  would  likely  have  in-depth  knowledge  about  real-time  performance  and 
how  to  analyze  for  performance. 

•  measurement  specialist:  develops  both  the  component  and  assembly  measurement  appara¬ 
tus  and  obtains  the  component  and  assembly  measurements.  This  person  must  know  how 
to  measure  and  empirically  validate  the  property  theories  in  the  PECT  and  would  typically 
have  broad  knowledge  of  statistical  methods. 

•  PECT  designer:  unifies  the  constructive  and  analysis  models  through  co-refinement  (as 
manifested  in  the  analysis  interfaces)  and  then  packages  and  deploys  the  PECT.  This  per¬ 
son  holds  the  holistic  view  of  the  PECT  and  communicates  with  the  other  workers  to  keep 
the  various  parts  of  the  PECT  development  synchronized. 
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In  addition  to  those  workers  actually  involved  in  the  PECT  process,  two  other  roles  are  key  to 
its  successful  execution: 

•  system  specialist:  has  in-depth  knowledge  about  the  execution  environment  that  usually 
affects  the  behavior  of  the  components  residing  in  the  PECT.  For  example,  the  PECT  may 
need  to  be  designed  to  restrict  the  use  of  the  operating  system  to  meet  the  set  goals. 

•  domain  expert:  knows  how  the  PECT  will  be  used  from  the  end  customer’s  perspective 
and  provides  input  about  existing  standards,  methods,  and  user  profiles.  This  role  is 
important,  because  it  helps  to  ensure  that  the  PECT’s  design  will  fit  the  system’s  intended 
usage. 

4.2  Workflows 

The  workflows  in  this  section  are  elaborations  of  the  PECT  overview  workflow  shown  in  Fig¬ 
ure  10  on  page  26. 


4.2.1  Definition  Phase 

The  workflow  shown  in  Figure  11  is  an  elaboration  of  the  Define  process  shown  in  Figure  10. 
The  Definition  phase  has  two  parallel  paths — one  to  define  the  functional  requirements  and 
another  to  gather  the  necessary  requirements  for  the  property  of  interest  and  set  the  PECT’s 
prediction  goals. 


Figure  11:  Definition  Phase  Workflow 
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Statistical  goals  should  be  based  on  the  business  need.  Because  producing  it  takes  effort,  a 
PECT  should  be  only  as  general  as  necessary  to  fill  that  need. 


4.2.2  Co-Refinement  Phase 

During  the  Co-Refinement  phase  shown  in  Figure  12,  we  define  the  constructive  and  analysis 
viewtypes,  and  the  interpretation  that  integrates  them.  We  also  define  a  plausible  and  possibly 
high-level  scheme  for  validating  predictions  based  on  the  analysis  viewtype. 

The  workflow  depicted  suggests  a  process  that  is  somewhat  more  structured  than  what  occurs 
in  actual  practice.  The  constructive  and  analysis  viewtypes  are  depicted  as  being  developed  in 
parallel,  but  they  are,  in  fact,  mutually  constraining,  as  explained  and  illustrated  in  Chapter  5. 
In  our  experience,  considering  validation  places  useful  constraints  on  the  development  of  the 
analysis  viewtype,  so  we  placed  “Define  Validation  Concept”  at  the  same  level  of  activity  in 
the  workflow  as  “Define  Constructive  Viewtype”  and  “Define  Analysis  Viewtype.”  A  similar 
argument  can  be  made  for  placing  “Define  Interpretation”  alongside  “Define  Validation  Con¬ 
cept,”  although  our  experience  suggests  that  defining  the  interpretation  is  usually  the  closing 
step  in  a  co-refinement  iteration. 


Figure  12:  Co-Refinement  Workfiow 
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The  iteration  within  the  co-refinement  workflow  must  not  be  confused  with  the  iteration  that 
results  from  the  empirical  validation  shown  in  Figure  13.  Within  co-refinement,  the  condition 
for  iteration  is  an  assessment  of  whether  the  resulting  constructive  assembly  is  sufficiently 
general  to  handle  an  anticipated  range  of  assemblies  and  whether  the  property  theory  underly¬ 
ing  the  analysis  viewtype  will,  with  reasonable  computation  and  interpretation  effort,  yield 
predictions  for  all  these  assemblies. 

4.2.3  Validation  Phase 

After  co-refinement,  the  resulting  PECT  is  validated  against  its  requirements.  The  definition 
workflow  shown  in  Figure  11  identifies  functional  requirements  and  tolerance  requirements. 
Functional  requirements  define  the  range  of  assemblies  that  must  be  “spanned”  by  the  con¬ 
structive  model  and  its  interpretations;  the  tolerance  requirements  refer  to  the  desired  quality 
of  predictions  based  on  the  analysis  viewtypes  supported  by  the  PECT.  The  workflow  shown 
in  Figure  13  is  biased  toward  the  validation  of  analysis  viewtypes  whose  underlying  property 
theories  are  empirical  (i.e.,  based  in  measurement)  rather  than  formal  (i.e.,  based  in  logic).  As 
this  workflow  is  currently  defined,  it  addresses  the  validation  of  both  functional  requirements 
and  statistical  goals. 
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Figure  13:  Empirical  Validation  Workflow 

There  are  four  general  aspects  to  carrying  out  an  empirical  validation  study: 

1 .  Define  the  validation  goal. 

The  goal  is  a  normative  or  informative  statement  about  the  predictive  power  of  the  PECT. 
Typically,  a  goal  is  stated  in  terms  of  the  probability  that  a  prediction  will  lie  within  some 
accuracy  bounds,  with  some  stated  confidence  level. 

2.  Define  the  validation  process. 

Validating  the  accuracy  of  a  prediction  is  analogous  to  validating  a  scientific  theory.  That 
is,  behaviors  are  observed  systematically  under  controlled  circumstances,  and  this 
observed  behavior  is  then  compared  to  predicted  behavior.  The  key  word  is  systematic, 
since  validation  results  should  be,  above  all  else,  objective  and  hence  repeatable. 
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3.  Collect  validation  data. 

This  step  involves  conducting  the  actual  validation  experiment.  Important  in  this  activity 
are  good  laboratory  techniques — for  example,  keeping  good  notes  and  records  so  that 
anomalies  can  be  studied  and  results  can  be  repeated. 

4.  Analyze  the  results. 

The  purpose  of  the  analyses  is  twofold:  first,  to  objectively  and  reliably  describe  the  pre¬ 
dictive  powers  of  a  PECT;  second,  to  provide  analysis  data  to  support  additional  co-refine¬ 
ment,  should  normative  goals  fail  to  be  satisfied. 

This  workflow  is  expanded  considerably  in  Chapter  6. 


4.2.4  Packaging  Phase 

The  workflow  for  packaging  is  a  lacuna  at  this  time.  The  objectives  of  packaging  are  twofold. 
One  objective  is  to  ready  the  technology  for  deployment,  including  the  development  of  instal¬ 
lation  support,  documentation,  and  so  forth.  A  second  and  more  fundamental  objective  is  to 
design  automation  support  for  the  PECT  to  minimize  the  property-theory-specific  expertise 
required  by  PECT  users  to  make  effective  use  of  analysis  viewtypes  supported  by  the  PECT. 
This  area  requires  further  work. 
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5  Co-Refinement  of  the  Controller  PECT 


5.1  What  Is  Co-Refinement? 

PECT  creation  is  a  process  of  iterative  negotiation  between  the  constructive  and  analysis 
points  of  view.  We  call  this  process  co-refinement.  Co-refinement  does  not  necessarily  follow 
a  rigorously  defined  process,  but  rather  it  is  channeled  by  a  set  of  forces  that  guide  co-refine¬ 
ment  as  the  two  points  of  view  (models)  converge  into  a  PECT. 

The  constructive  point  of  view  pushes  for  assembly  generality;  the  more  assemblies  that  can 
be  represented  in  the  constructive  model,  the  better.  The  analysis  point  of  view  pushes  for  pre¬ 
dictability,  which  implies  constraining  the  constructive  model  to  adhere  to  the  assumptions  of 
the  property  theories  that  enable  analysis  and  prediction.  These  forces  tend  to  act  in  opposition 
to  each  other. 

Practicability,  tractability,  and  interpretability  also  weigh  in  as  important  forces  and  act  as 
moderators.  Practicability  pushes  for  a  level  of  generality  that  is  necessary  (but  no  more  than 
necessary)  to  represent  some  class  of  realistic  system;  it  acts  as  a  bounds  on  generality.  Tracta¬ 
bility  is  concerned  with  the  level  of  difficulty  in  solving  the  analysis  model;  it  acts  as  a  bounds 
on  predictability.  For  example,  it  might  not  be  viable  to  require  a  supercomputer  to  solve  the 
analysis  model.  Interpretability  ensures  that  automatic  translation  from  the  constructive  model 
to  the  analysis  model  is  possible.  This  possibility  implies  the  ability  to  formally  specify  both 
models  and  to  translate  the  constructive  representation  into  the  analysis  representation.  Vali- 
datability  is  concerned  with  the  level  of  difficulty  in  obtaining  empirical  validation  of  the  anal¬ 
ysis  model. 


5.2  How  Co-Refinement  Proceeds 

The  co-refinement  process  starts  with  an  initial,  not  necessarily  formal,  description  of  the  “lan¬ 
guages”  for  the  constructive  and  analysis  models.  The  elements  of  the  constructive  model  lan¬ 
guage  are  influenced  by  the  PECT’s  target  domains  and  how  systems  in  those  domains  will  be 
structured,  developed,  deployed,  and  sustained  (evolved)  over  time.  The  elements  of  the  anal¬ 
ysis  model  language  are  simply  a  subset  of  some  property  theory  that  is  suitable  for  analyzing 
the  behavior  of  interest. 
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Initially,  each  model  is  independently  subjected  to  its  own  starting  criteria.  The  constructive 
model has  to  be  expressive  enough  to  describe  the  types  of  components  and  interactions 
expected  in  the  domains  that  will  use  the  PECT.  However,  initially,  the  constructive  model 
might  not  be  interpretable  vis-a-vis  the  analysis  model,  and  the  analysis  model  might  not  be 
general  enough  to  account  for  all  aspects  of  the  constructive  model. 

Given  a  starting  point  for  the  constructive  and  analysis  models,  the  forces  guide  co-refinement 
as  the  initial  models  converge  into  final  models  that  define  the  PECT.  While  there  is  no  precise 
set  of  steps  for  determining  the  sequence  of  intermediaries  from  the  initial  models  to  the  final 
models,  there  are  some  rules  of  thumb. 

Generality  increases  monotonically.  After  the  first  iteration,  the  PECT  designer  should  have 
reasonable  confidence  that  the  constructive  model  is  interpretable  and  that  the  analysis  model 
is  tractable.  Therefore,  the  dominant  force  in  co-refinement  at  this  point  is  moving  the  con¬ 
structive  model  to  the  desired  state  of  generality.  The  only  reason  to  reduce  generality  would 
be  if  the  designer’s  confidence  in  interpretability  proved  to  be  unwarranted. 

Interpretability  increases  monotonically.  While  co-refinement  might  not  start  with  interpret¬ 
ability,  it  must  end  with  it.  This  leads  us  to  believe  that  interpretability  must  monotonically 
increase  as  co-refinement  proceeds.  This  means  that,  during  co-refinement,  the  languages 
describing  the  constructive  and  analysis  models,  and  the  interpretation  rules  become  more  for¬ 
mal.  In  addition,  a  translator  for  the  constructive  model  is  developed  iteratively  as  co-refine¬ 
ment  proceeds. 

Validatability  is  an  invariant.  Validatability  is  a  defining  characteristic  of  a  PECT.  Unlike 
generality  and  interpretability,  however,  it  is  important  that  all  analysis  models  be  validatable, 
at  least  in  principle,  for  each  iteration  of  co-refinement.  By  in  principle,  we  mean  that  there 
must  be  a  plausible  means  for  obtaining  or  validating  component  properties,  and  for  falsifying 
predictions  made  in  the  analysis  model.  While  plausibility  is  not  an  objective  quality,  it  is  one 
that  has  a  reasonable  intuitive  meaning. 

Using  these  rules,  co-refinement  drives  for  increasing  generality  in  the  constructive  model 
until  all  the  following  stopping  conditions  are  met: 

•  The  practicability  boundary  is  reached — striving  for  more  generality  is  not  worth  it. 

•  The  interpretability  boundary  is  reached — an  automatic  translation  becomes  impossible. 

•  The  tractability  boundary  is  reached— because  of  its  complexity,  solving  the  analysis 
model  is  too  costly. 

This  iterative  scheme  was  depicted  in  Figure  12  on  page  30. 


17.  The  terms  constructive  model  and  constructive  model  language  are  used  interchangeably  in  this  report. 
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5.3  Key  Ideas  for  the  Controller  PECT 

Performance  is  important  in  the  intended  domain  for  this  PECT— power  station  control.  Sen¬ 
sor  inputs  are  gathered  at  various  rates,  inputs  are  coalesced,  and  actions  are  taken  depending 
on  the  input  values.  Computations  must  adhere  to  both  worst-case  and  average-case  latency 
requirements.  Typically  systems  are  multiprocessing  on  a  single  processor  or  multiprocessors, 
connections  can  be  both  synchronous  and  asynchronous,  and  some  data  stores  must  be  updated 
atomically. 

The  property  theory  for  the  controller  PECT  is  based  on  rate  monotonic  analysis  (RMA) 
[Klein  93].  RMA  is  well  suited  for  predicting  worst-case  latency  for  a  collection  of  processes 
on  a  single  processor  in  a  mostly  (but  not  solely)  deterministic  situation;  events  occur  mostly 
periodically,  and  process  execution  times  are  bounded  and  do  not  vary  dramatically.  RMA 
requires  information  such  as  process  periods  and  execution  times  and  priorities;  whether 
mutually  exclusive  access  to  shared  resources  is  needed;  whether  there  are  intervals  of  nonpre- 
emptability;  and  whether  each  process  has  a  unique  priority.  In  general,  RMA  strives  to 
account  for  any  factor  that  can  contribute  to  latency — the  time  interval  from  when  an  event 
occurs  until  it  has  completely  been  processed. 


5.4  Co-Refinement  of  the  Controller  PECT 

This  section  describes  the  starting  state  (initial  condition)  for  each  model  and  then  walks 
through  the  five  iterations  of  the  co-refinement  process,  leading  to  ^he  property  theory 
whose  validation  is  the  subject  of  Appendix  B.  For  each  iteration,  constraints  on  the  constmc- 
tive  model  and  model  semantics  (i.e.,  what  is  predicted)  are  described.  The  five  iterations  are 

•  predicting  worst-case  latency  (X^v) 

•  predicting  average-case  latency  (X^) 

•  introducing  mutual  exclusion  and  worst-case  blocking  (X^g) 

•  predicting  average-case  blocking  (Xy^g) 

•  introducing  asynchrony  (X^gy^) 

It  should  be  noted  that  this  is  an  a  posteriori  examination  of  a  co-refmement  experience.  A  pri¬ 
ori,  you  would  not  expect  to  know  the  specifics  of  each  iteration  or  how  many  iterations  are 
required. 

5.4.1  Initial  Conditions 

The  initial  constructive  model  was  relatively  simple.  It  consisted  of  a  collection  of  compo¬ 
nents,  where  each  component  had  a  set  of  source  and  sink  pins.  Since  concurrency  was  known 
to  be  important,  each  component  could  have  either  no  threads  or  one  thread  associated  with 
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it. Since  some  components  had  to  be  activated  periodically,  there  was  a  special  source  pin 
that  was  denoted  as  periodic.  Naturally,  these  components  were  threaded.  Sink  pins  could  be 
either  reentrant  or  non-reentrant.  Both  synchronous  and  asynchronous  connections  were 
allowed. 

The  analysis  model  is  based  on  a  collection  of  real-time  analysis  techniques  known  as  RMA 
for  predicting  worst-case  latency.  Latency  is  the  measure  of  how  long  it  takes  to  service  an 
event.  For  example,  we  consider  reaching  a  voltage  sensor’s  read  time  as  the  occurrence  of  an 
event.  Servicing  the  event  comprises  all  the  computations  that  are  necessary  to  respond  to  the 
event,  such  as  acquiring  input  from  the  sensor,  converting  the  input  to  a  number  with  some 
physical  meaning,  filtering  noise,  combining  the  input  with  other  data,  testing  the  data  against 
some  important  physical  conditions,  and  then  issuing  some  output.  These  computations  take 
place  on  a  single  processor  that  is  also  servicing  other  events.  Latency  is  how  long  it  takes 
from  the  voltage  sensor’s  read  time  (not  necessarily  when  the  sensor  is  actually  read)  to  the 
completion  of  the  response  to  the  event. 

The  initial  analysis  model  used  a  subset  of  the  modeling  capabilities  of  RMA  including 

•  considering  only  periodic  events,  that  is,  those  that  occur  at  regular  intervals 

•  each  periodic  event  is  allocated  its  own  process  (or  thread) 

•  each  process  is  assumed  to  have  a  constant,  unique  priority 

•  the  variability  of  the  execution  time  for  any  given  component  is  bounded 

•  the  worst-case  latency  of  all  events  is  less  than  or  equal  to  the  period 

•  only  reentrant,  synchronous  connections  are  allowed 

•  no  blocking  is  allowed  (sources  of  blocking  include  critical  sections  and  nonpreemptable 
sections) 


5.4.2  First  Iteration:  Worst-Case  Latency  (Xw) 

Tractability  and  interpretability  were  the  dominant  forces  during  this  iteration.  They  caused  us 
to  both  extend  the  constmctive  model  with  the  annotations  necessary  to  provide  the  analysis 
model  with  the  needed  information  and  to  constrain  the  constructive  model  to  conform  to  the 
restrictions  of  the  initial  conditions  of  the  analysis  model.  The  annotations  included 

•  specifying  execution  times  as  properties 

•  specifying  unique  priorities  for  each  thread.  This  turned  out  to  be  a  significant  restriction, 
because  Windows  imposed  severe  limitations  on  the  number  of  priority  levels. 


18.  This  has  since  been  generalized.  See  Section  3.3.1 . 
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Modifications  to  the  constructive  model  involved  mainly  the  thread  model.  Since  periodic 
invocations  were  important,  we  decided  to  extend  the  constructive  model  with  an  explicit 
notion  of  time  using  an  environment  corhponent,  the  clock  component.  This  extension  elimi¬ 
nated  the  need  for  a  periodic  source  pin.  Also,  since  the  analysis  model  was  not  considering 
blocking  during  this  iteration,  we  disallowed  threaded  components,  because  they  would  have 
resulted  in  non-reentrant  components  and  hence  blocking.  The  constructive  model  was  also 
constrained  to  allow  only  synchronous  calls,  conforming  to  this  restriction  of  the  analysis 
model. 


With  these  restrictions  to  the  analysis  model,  the  worst-case  latency  for  each  process  could  be 
calculated  using  the  following  fixed-point  calculation; 


■"n  +  1 


s 

j  =  l 


T; 


Cj  +  Cj 


This  formula  can  be  used  to  compute  the  worst-case  latency  Lj  for  the  ith  process.  Cj  is  the 
execution  time  of  the  tth  process;  Tj  is  the  period  of  the  ith  process.  Processes  1  through  i-1  are 
assumed  to  be  all  the  processes  whose  priorities  are  higher  than  that  of  process  i.  The  iterative 
calculation  starts  by  using  Cj  as  the  first  guess  for  the  worst-case  latency  by  setting  Lj  to  Cj. 
Then  it  computes  Ln+j  using  the  formula  above.  It  continues  until  Ln  =  at  which  point 
the  fixed  point  has  been  reached,  and  Ln^.j  is  the  worst-case  latency. 

The  interpretation  from  the  constructive  model  to  this  analysis  model  (the  above  formula)  is 
straightforward.  Execution  time  (associated  with  sink  pins),  periods  (associated  with  clock 
components),  and  priority  (associated  with  threads)  all  map  directly  to  the  formula.  The  only 
complication  occurred  for  the  case  in  which  a  clock  component  initiates  a  sequence  of  compo¬ 
nent  calls,  as  shown  in  Figure  14. 


Figure  14:  Clock  Component  Initiating  a  Sequence  of  Interactions 

In  this  case,  the  period  of  the  clock  corresponds  directly  to  the  period  of  process  i  in  the  analy¬ 
sis  model,  the  priority  of  component  Cj  corresponds  to  the  priority  of  process  i  in  the  analysis 
model,  and  the  sum  of  the  execution  times  of  components  Cj,  C2,  C3,  and  so  on,  corresponds  to 
the  execution  time  of  process  i. 
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At  this  point,  we  did  not  yet  have  an  automatic  translation  from  the  constructive  to  the  analysis 
model,  but  we  did  have  a  high  degree  of  confidence  that  such  a  translation  could  be  created.'^ 


5.4.3  Second  Iteration:  Average  Latency  (x^) 

For  the  second  iteration,  practicability  started  to  exert  influence  mainly  in  terms  of  the  need  of 
the  constructive  model  to  satisfy  average-case  latency  requirements  in  addition  to  worst-case 
latency  requirements.  Another  facet  of  practicability  influencing  this  iteration  was  the  realiza¬ 
tion  that  the  zero-phasing^®  assumption  of  the  analysis  model  was  not  realizable  in  the  con¬ 
structive  model.  Therefore  during  this  iteration,  the  constructive  model  did  not  change;  the 
analysis  model  evolved  to  predict  average-case  latency  and  to  handle  non-zero  phasing. 

Zero  phasing  is  significant  because  a  key  idea  of  RMA  is  that  worst-case  latency  occurs  for 
process  i  when  it  becomes  ready  to  execute  at  exactly  the  same  time  that  all  higher  priority 
processes  do.  The  fixed-point  formula  shown  in  the  previous  section  computes  worst-case 
latency  assuming  zero-phasing.  If  the  constructive  model  does  not  in  fact  adhere  to  zero  phas¬ 
ing,  the  prediction  might  produce  a  result  that  is  too  pessimistic.  Accounting  for  non-zero 
phasing  requires  only  minor  changes  to  the  fixed-point  formula  shown  in  the  previous  section. 
These  changes  were  made  during  this  iteration. 

A  more  substantive  change  to  the  analysis  model  was  required  to  calculate  average  latency. 
The  critical  observation  was  that  the  execution  profile  of  process  i  repeats  itself.  The  interval 
of  repetition  (known  as  the  hyper-period)  is  equal  to  the  least  common  multiple  (LCM)  of  Ti, 
T?, ...,  T; .  Every  NP  period,  process  i’s  performance  behavior  repeats,  where  NP,-  =  LCM(T|, 
T2, ...» T,)/T,.  Therefore,  the  latency  for  the  first  instance  of  process  i  should  be  the  same  as  the 
latency  for  the  l-fNP/**  instance.  To  calculate  the  average  latency  for  process  i,  we  simply 
compute  the  average  of  the  NP,-  instances  of  process  i  during  a  hyper-period.  The  fixed-point 
calculation  above  had  to  be  generalized  to  work  across  the  hyper-period  instead  of  just  stop¬ 
ping  after  the  first  instance  of  the  job. 

At  this  point,  we  performed  a  “spot  validation”  of  the  interpretation  and  analysis  model.  That 
is,  we  constructed  a  small  experiment,  consisting  of  a  smalt  sample  (<  5  simple  assemblies)  to 
see  if  we  were  predicting  average  latency  accurately.  The  results  were  encouraging. 


19.  Although  rewrite  rules  were  possible,  we  chose  not  to  Implement  them,  because  the  generality  of  this  approach  to  interpre¬ 
tation  was  not  clear. 

20.  We  say  two  or  more  events  are  zero-phased  in  time  when  they  happen  at  the  same  moment  (i.e.,  there  Is  no  time  delay 
between  them).  In  the  context  of  the  controller  PECT,  zero-phasing  means  that  process  /  will  become  ready  to  execute  at 
the  exact  moment  alt  higher  priority  processes  become  ready  to  execute  (the  critical  instant). 
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5.4.4  Third  Iteration:  Worst-Case  Latency  +  Blocking  (%b) 

During  this  iteration,  practicability  continued  to  be  a  dominant  influence.  The  controller 
domain  required  non-reentrant  execution  within  components  to  ensure  data  consistency. 

During  the  first  two  iterations,  the  only  influences  on  process  latency  (in  theory)  were  its  own 
execution  time  and  preemption  from  higher  priority  processes.  With  non-reentrancy,  a  higher 
priority  process  might  have  to  wait  while  a  lower  priority  process  is  executing  a  non-reentrant 
component.  This  effect  is  known  as  blocking.  It  prolongs  latency  and  must  be  accounted  for  in 
the  analysis  model. 

To  account  for  blocking  in  the  analysis  model,  a  blocking  term  is  simply  added  to  the  expres¬ 
sion  on  the  right  side  of  the  worst-case  latency  equation  shown  on  page  39.  However,  the  real 
issue  is  how  to  control  the  magnitude  of  this  term.  To  do  so,  we  impose  a  restriction  on  the 
constructive  model  that  is  based  on  another  key  result  of  RMA.  One  of  the  synchronization 
protocols  discovered  by  RMA  is  the  priority  ceiling  protocol.  That  protocol  exhibits  an  impor¬ 
tant  property  known  as  the  blocked-at-most-once  property,  which  says  that  a  process  that  exe¬ 
cutes  in  several  critical  sections  can  be  blocked  in,  at  most,  one  critical  section. 

The  priority  ceiling  protocol  requires  that  when  more  than  one  component  interacts  with  a 
non-reentrant  sink  pin,  the  thread  for  that  sink  pin  must  execute  at  a  priority  level  that  is  at 
least  as  high  as  the  priority  of  the  effective  thread  of  any  of  the  calling  components.  This  prior¬ 
ity  level  is  known  as  the  ceiling  priority.  Note  that  for  this  priority  assignment  to  emulate  the 
priority  ceiling  protocol,  the  thread  that  is  assigned  this  priority  cannot  suspend  for  any  reason, 
including  input/output  (I/O). 

The  prediction  obtained  using  the  priority  ceiling  is  a  worst-case  prediction  with  respect  to 
blocking,  because  a  blocking  term  is  used  for  every  job  in  the  hyper-period,  regardless  of 
whether  the  job  is  actually  blocked  during  its  execution.  Assemblies  must  honor  the  priority 
ceiling  to  be  well  formed  with  respect  to  Since  the  constructive  model  has  no  concept  of 
priority,  it  is  the  interpretation  rather  than  the  constructive  model  that  enforces  this  topological 
constraint. 


5.4.5  Fourth  Iteration:  Average  Latency  +  Blocking  (x^b) 

The  goal  of  this  iteration  is  to  calculate  average  latency  in  the  presence  of  blocking,  taking  into 
account  the  possibility  that  blocking  will  occur  only  for  a  portion  of  the  jobs  in  a  hyper-period. 
The  initial  challenge  for  this  iteration  was  deriving  a  variation  of  the  fixed-point  calculation 
that  could  also  determine  when  blocking  did  and  did  not  occur.  To  make  this  determination, 
the  notion  of  a  subtask  was  introduced.  A  subtask  is  a  portion  of  a  component’s  computation 
distinguished  by  its  priority;  different  subtasks  have  different  priorities. 
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At  this  point,  a  purely  analytic  approach  became  unwieldy — in  fact,  nearly  intractable.  How¬ 
ever,  the  analysis  model  yielded  to  a  hybrid  simulation-based  approach  [Schwetman  78].  In  a 
hybrid  simulation  approach,  an  analysis  model  is  used  to  compute  values  that  define  initial  or 
test  conditions  in  a  simulator.  Details  of  the  hybrid  simulation  are  provided  in  Appendix  A. 

At  this  stage,  a  formal  interpretation  was  defined.  Although  we  knew  that  the  average-case 
latency  theories  were,  in  principle,  validatable,  it  was  by  no  means  clear  that  the  earlier  spot 
validation  would  scale  from  to  X^b-  The  PECT  was  not  yet  sufficiently  general,  since  it  did 
not  accommodate  assemblies  with  asynchronous  pins.  This  led  to  the  final  iteration. 


5.4.6  Fifth  Iteration:  Asynchronous  Interactions  (x^ba) 

The  previous  iterations  had  all  involved  adding  new  terms  and  features  to  the  analysis  model, 
and  introducing  or  removing  constraints  on  the  constructive  model  to  accommodate  those 
additions.  At  this  point,  it  was  clear  that  the  final  and  necessary  step  of  the  co-refinement  pro¬ 
cess  would  be  to  relax  the  prohibition  against  asynchronous  pins.  It  was  unclear  at  this  point 
how  this  constraint  could  be  relaxed,  or  how  much  its  relaxation  would  affect  the  analysis 
model  or  simulation.  Our  great  fear  was  that  it  would  result  in  some  kind  of  fundamental  dis¬ 
continuity,  effectively  demonstrating  that  the  basic  RMA  approach  was  insufficient  or  vastly 
complicating  the  simulator  (loss  of  tractability). 

Our  fears  proved  to  be  unfounded,  however.  We  made  the  fortuitous  discovery  that  we  could, 
through  the  application  of  rewrite  rules,  transform  a  constructive  assembly  with  an  arbitrary 
topology  of  asynchronous  interactions  into  a  behaviorally  equivalent  one  (from  the  perspec¬ 
tive  of  latency  computation)  with  a  more  restrictive  topology,  as  with  synchronous  pins  only. 
These  restrictions  simplified  the  interpretation  and  preserved  the  hybrid  simulation  model. 
Thus,  we  were  able  to  introduce  asynchrony  to  the  latency  model  strictly  through  rewriting 
and  interpretation. 
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6 


Empirical  Validation 


6.1  What  Is  Empirical  Validation? 

There  are  two  classes  of  property  theory:  those  based  in  logic  and  those  based  in  tneasurement. 
Logical  property  theories,  such  as  those  found  in  model  checking  and  other  forms  of  program 
verification,  involve  demonstrative  reasoning:  the  validity  of  a  property  is  mathematically 
demonstrated,  or  proven,  in  a  way  that  bars  contradiction.  Empirical  property  theories,  such  as 
^ABA’  involve  plausible  reasoning:  the  validity  of  a  property  theory  is  established  inductively, 
based  on  observations. 

The  fact  that  empirical  theories  are  based  on  observations  and  measurement  introduces  uncer¬ 
tainty.  The  property  theory  itself  may  abstract  aspects  of  an  implementation  that  influence  the 
assembly-level  property.  This,  in  turn,  introduces  variability  and  some  degree  of  non-determi- 
nacy,  for  we  cannot  give  a  systematic  accounting  of  an  abstracted  system  aspect  without  undo¬ 
ing  the  effect  of  its  abstraction. 

Empirical  validation  is  the  means  by  which  we  validate  empirical  property  theories.  The  vali¬ 
dation  does  not  yield  a  simple  “yes”  or  “no.”  Instead,  it  produces  statistical  evidence  of  a  prop¬ 
erty  theory’s  effectiveness  and  of  the  component  property  measures  on  which  the  validation’s 
predictions  are  based.  Before  describing  the  workflow  for  empirical  validation  in  Section  6.3, 
we  provide  a  brief  overview  of  the  statistical  models  used  to  express  statistical  evidence. 


6.2  Introduction  to  Statistical  Labels 

Rigorous  observation  takes  the  form  of  measurement,  and  measurement  invariably  introduces 
error.  Thus,  measured  component  properties  are  described  not  as  discrete  values,  but  as  proba¬ 
bility  density  functions  over  a  range  of  values.  As  was  the  case  with  component  properties, 
measured  assembly  properties  are  described  using  probability  density  functions  over  a  range 
of  values.  Likewise,  determining  the  effectiveness  of  a  property  theory  also  involves  measure¬ 
ment,  this  time  in  a  more  classical  application  of  the  scientific  method.  The  property  theory  is 
a  hypothesis;  it  is  tested  by  comparing  assembly  properties  predicted  by  the  hypothesized 
property  theory  with  observed  assembly  properties. 

We  refer  to  the  statistical  descriptors  of  component  properties  and  property  theory  effective¬ 
ness  as  statistical  labels.  Our  position  is  also  that  all  empirical  properties  and  property  theories 
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can  be  described  in  a  uniform  way,  using  the  same,  or  very  similar,  statistical  models,  resulting 
in  standard  labels. 


6.2.1  Component  Labels 

The  measurement  and  subsequent  description  of  empirical  component  properties  do  not  intro¬ 
duce  novel  methodological  challenges.  Objective  measurement  requires  a  measurement  object 
(the  component),  a  measurement  scale  for  the  component  property  of  interest  (e.g.,  time,  in 
seconds),  and  a  measurement  apparatus  (e.g.,  the  Microsoft  Windows’  high-performance 
clock).  Measurement  is  conducted  using  the  apparatus  within  some  controlled  environment 
and  with  control  exerted  over  all  independent  variables  of  the  dependent  (measured)  property. 


Component  Property  Estimators 

In  general,  the  measurand  Y,  or  property  of  interest,  is  a  function  of  N  values: 

Y  =  f(Xj,  X2, ...,  Xfj)  [Taylor  94].  For  latency,  these  values  include  execution  time,  blocking 
time,  and  period.  We  would  like  to  know  the  true  value  of  Y,  component  latency.  Of  course,  the 
true  value  is  not  obtainable,  as  the  following  definition  makes  clear: 

True  Value:  the  mean  (p.)  that  would  result  from  an  infinite  number  of  measure¬ 
ments  of  the  same  measurand  carried  out  under  repeatable  conditions,  assuming 
no  systematic  error. 

Because  we  cannot,  even  in  principle,  know  the  true  value  of  p,  we  must  use  an  estimator  for 
it,  one  that  is  produced  by  statistical  methods. 

For  example,  we  take  sample  observations  of  X  and  use  their  average  x  as  the  estimator  of  p,  a 
population  parameter.  The  uncertainty  associated  with  this  estimator  is  expressed  as  the  devia¬ 
tion  s  such  that  the  true — and  unknown — value  of  p  will  fall  within  some  interval  x±ks  with 
some  specified  confidence.  The  factor  k  is  known  as  the  coverage  factor.  When  k=l,  x+ks 
yields  a  68%  confidence  interval  (one  standard  deviation).  That  is,  we  have  0.68  confidence 
that  this  interval  contains  p.  Typically,  we  compute  the  0.95  confidence  interval  (k=2),  which 
yields  higher  confidence  but  a  larger  bound.  This  confidence  interval  expresses  a  fundamental 
component  measure;  it  is  fundamental  because  x±ks  is  how  a  component  property  is  modeled 
in  any  property  theory  parameterized  by  that  component  property. 


Component  Property  Intervals 

It  might  be  useful  to  have  other  descriptors,  or  labels,  for  component  properties.  One  such 
label  is  the  tolerance  interval.  For  example,  in  the  case  of  component  latency,  the  consumer 
might  want  to  know  which  latency  interval  ±x  ms  contains  a  specified  proportion  p  of  all  exe¬ 
cutions  of  component  c.  For  instance,  a  tolerance  interval  with  p=0.95  might  state  that  there  is 
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a  0.95  probability  that  any  given  component’s  execution  will  have  a  latency  of  50ms±l7ms , 
where  ±17ms  is  the  computed  tolerance  interval. 

Another  potentially  useful  label  is  the  conceptual,  but  not  mathematical,"'  inverse  of  the  above 
tolerance  interval.  This  label  is  the  confidence  interval  on  the  probability  of  satisfying  some 
property  specification.  Conceptually,  this  label  is  the  inverse  of  the  tolerance  interval,  since  we 
specify  the  latency  interval  here  and  use  it  to  compute  the  probability  that  any  particular  exe¬ 
cution  will  fall  within  this  interval.  For  example,  we  might  specify  the  latency  interval  ±5ms 
and  use  it  to  calculate  that  there  is  a  0.64  probability  that  any  given  component’s  execution 
will  fall  within  the  interval  50ms±5ms. 


6.2.2  Property  Theory  Labels 

Measuring  and  describing  the  effectiveness  of  property  theories  is  methodologically  more 
challenging  than  measuring  component  properties.  As  noted  earlier,  at  the  limit,  this  measure¬ 
ment  process  reduces  to  theory  falsification  as  it  is  practiced  in  the  empirical  sciences;  in  that 
sense,  at  least,  no  new  ground  is  broken.  Our  concern  is  with  characterizing  (labeling)  the 
quality  of  a  property  theory.  Such  a  characterization  translates  into  labeling  the  accuracy  of  the 
theory’s  predictions  and  determining  how  often  they  are  accurate. 


One-Tail  Inferential  Intervals 

We  use  inferential  statistical  models  to  characterize  how  effective  a  property  theory  is  likely  to 
be  for  future  predictions.  For  this  purpose,  we  use  confidence  and  tolerance  intervals.  For 
property  theories,  we  are  usually  interested  in  the  MRE  between  predicted  and  observed  val¬ 
ues.  For  latency,  this  equation  is  used: 

a.A 

where  aX  is  the  measured  assembly  latency  and  aX  is  the  predicted  latency.^^  A  normative 
confidence  interval  will  describe  the  probability  that  the  MRE  for  a  particular  prediction  will 
lie  within  a  specified  MRE  interval. 

Notice  that  the  lower  bound  for  an  MRE  interval  represent  situations  where  predictions  are 
better  than  the  mean  MRE  of  the  statistical  sample  (i.e.,  in  our  experiment,  for  N=30  assem¬ 
blies).  While  we  might  want  to  know  how  frequently  our  predictions  are  better  than  the  mean, 
we  might  be  concerned  only  when  the  predictions  are  worse  than  the  mean.  In  this  case,  we 


21 .  Different  equations  are  used  to  compute  conceptual  and  mathematical  intervals. 

22.  We  note  in  passing  that  a.X'  is  described  using  the  fundamental  label  for  components.  That  is,  for  statistical  analysis,  the 
assembly  is  treated  as  a  component,  and  the  estimator  for  assembly  latency  Is  described  by  an  Interval  obtained  using  a 
coverage  factor  k=2. 
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use  a  one-tail  interval,  or  bound,  instead  of  a  two-tail  interval.  The  one-tail  tolerance  interval 
for  ^aba  summarized  in  Table  3. 


Table  3:  Distribution-Free  Tolerance  Interval  fork^^A 


Part  of  Interval 

Description 

N  =  65 

sample  size 

Y  =  0.9942 

confidence  level 

p  =  0.80 

population 

M^mre  =  0.04 

MRE 

UB  =  10% 

upper  bound 

The  tolerance  interval  for  interpreted  as  saying  that  80%  of  latency  predictions 

will  not  exceed  an  MRE  of  roughly  10%,  and  that  the  manner  in  which  that  interval  is  con¬ 
structed  yields  a  99%  confidence  that  this  upper  bound  is  correct."^  As  with  measures  of  com¬ 
ponent  properties,  one-  and  two-tail  normative  confidence  intervals  can  be  computed  by 
specifying  interval  bounds  and  computing  p,  the  probability  that  any  particular  prediction  will 
satisfy  these  specified  bounds. 

Linear  Correlation 

Linear  correlation  is  a  descriptive  statistic:  it  describes  the  strength  of  correlation  between  two 
data  sets  and  is  not  directly  useful  for  drawing  inferences  about  future  data  sets.  A  consumer 
might  be  interested  in  linear  correlation  analysis  as  a  descriptor  of  previous  experimental  vali¬ 
dations  of  a  property  theory’s  accuracy. 

We  characterize  that  accuracy  using  linear  correlation  analysis,  which  allows  us  to  assess  the 
strength  of  the  linearly  relation  between  two  variables — in  our  case,  predicted  and  observed 
assembly  latency.  The  result  of  such  analysis  is  the  coefficient  of  determination,  0  <  R“  <  1, 
where  0  represents  no  relation,  and  1  represents  a  perfect  linear  relation.  In  a  perfect  prediction 
model,  predicted  and  observed  latency  would  be  identical;  therefore,  the  goal  for  the  model 
builder  is  a  linear  relation. 

For  Xaba'  ^  distribution-free  linear  correlation  was  used  (Spearman  rank  correlation).  The 
resulting  R“  =  0.997  means  that  there  is  <  0.001  probability  that  the  reported  correlation  could 
have  been  achieved  by  chance. 


23.  As  discussed  in  Appendix  B,  this  is  a  pessimistic  interpretation  of  the  validation  data  that  includes  a  systematic  error  that 
can,  with  a  high  degree  of  certainty,  be  removed  from  X^ba- 
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6.3  Workflow  for  Empirical  Validation 

The  workflow  in  Figure  15  expands  the  first  two  steps  of  the  four-step  validation  process  illus¬ 
trated  in  Figure  13  on  page  32.  Figure  13’s  workflow  is  reproduced  in  the  left  portion  of  Figure 
15;  the  first  two  steps  of  Figure  13  have  been  replaced  by  their  expansions,  resulting  in  a  six- 
step  workflow. 


Data 


Define  Validation  Goal 


e  o 


Define  sampling  Develop  measurement 
procedure  infrastructure 


Figure  15:  Expanded  Workflow  for  Empirical  Validation 

In  the  remainder  of  this  section,  we  describe  the  six  steps  shown  in  Figure  15. 


6.3.1  Define  Validation  Goal 

The  overall  objective  of  empirical  validation  is  to  assign  a  statistical  label  to  a  property  theory 
that  describes  the  effectiveness  of  that  theory  and  to  design  the  means  of  producing  trusted 
component  labels  in  support  of  that  theory. 

Two  types  of  goals  yielding  different  types  of  statistical  labels  are  possible:  normative  and 
informative.  Normative  goals  express  a  prediction  requirement  that  has  to  be  met.  For  exam¬ 
ple,  predictions  might  be  required  to  be,  on  average,  accurate  to  within  0.5%.  An  informative 
goal  is  free  of  norms — it’s  up  to  the  validator  to  describe  the  effectiveness  of  predictions.  For 
example,  the  validator  might  compute  the  upper  bound  of  a  confidence  interval  for  60, 70,  80, 
and  90%  of  predictions,  assuming  varying  levels  of  confidence. 
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Part  of  defining  the  validation  outcome  involves  selecting  the  statistical  models  used  for 
labels.  As  discussed  in  Section  6.2.2,  we  have  used  the  MRE  as  a  basis  for  the  empirical  vali¬ 
dation  of  property  theories  and  two  forms  of  intervals  for  descriptive  and  inferential  labels. 
Those  choices  are  not  a  necessary  consequence  of  the  initial  “customer”  objective — rather 
they  are  choices  made  by  the  measurement  specialist  of  the  best  labels  for  a  particular  valida¬ 
tion  effort.  As  we  learn  more  about  empirical  validation,  we  anticipate  a  broadened  repertoire 
of  statical  models. 

For  X^BA'  design  problem  was  to  develop  a  controller  PECT  that  would  predict  latency 
with  an  upper  bound  of  a  tolerance  interval  for  MRE  <  0.05,  for  a  population  p  =  0.80,  with  a 
confidence  level  y  =0.99.  Thus,  the  empirical  validation  forXys^B^  was  dominated  by  normative 
goals. 


6.3.2  Define  Measures 

It  is  not  always  obvious  what  exactly  must  be  measured  to  validate  a  property  theory,  or  which 
measurement  units  are  best  to  express  those  measures.  Indeed,  frequently  there  are  several 
possible  measurement  units,  depending  on  whether  a  direct  or  indirect  approach  is  desired,  or 
on  the  degree  of  accuracy  or  stability  required.  A  good  discussion  of  measures  and  measure¬ 
ments  can  be  found  in  the  seminal  work  of  Fenton  and  Pleeger  [Fenton  97]. 

From  A,^bA’  have  identified  the  property  of  interest — latency — and  defined  it  as  the  total 
time  a  task  requires  to  perform  its  work.  The  constituents  of  this  total  time — execution  time 
and  blocking  time — also  had  to  be  expressed  in  terms  of  the  Pin  component  model  and  mea¬ 
surement  units.  We  also  had  to  know  whether  there  was  a  technical  basis  for  measurement  at 
all,  which,  surprisingly  perhaps,  was  not  trivial. 

Translating  the  notion  of  time  to  Pin  was  conceptually  straightforward.  Sink  and  source  pins 
provided  convenient  “hooks”  for  hanging  instrumentation.  Each  pin  can  be  instrumented  to 
record  the  moment  of  its  activation  (e.g.,  when  a  sink  pin  receives  a  request  or  when  a  source 
pin  sends  a  message  or  makes  a  request),  and  the  moment  of  its  deactivation  (e.g.,  when  the 
sink  pin  satisfies  its  request  or  when  a  source  pin  completes  its  message  send  or  has  its  request 
satisfied).  These  four  measurement  points  permit  a  variety  of  time  durations  to  be  recorded  on 
any  assembly. 

Defining  the  appropriate  units  of  time  depends  on  the  available  instruments.  For  X^i^bA’ 
chose  to  use  the  Microsoft  high-performance  clock,  which  records  times  in  nanoseconds. 
Before  making  this  selection,  though,  we  had  to  establish  that  that  clock  could  be  used  to  mea¬ 
sure  component  latency.  To  do  so,  we  had  to  first  demonstrate  that  the  latency  of  a  process,  as 
recorded  by  the  clock,  was  equal,  within  measurement  tolerance,  to  the  sum  of  system  time 
and  wait  time  for  a  process,  as  recorded  by  the  central  processing  unit  (CPU)  clock.  Once  we 
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had  established  this  correlation,  which  itself  required  a  validation  activity,  we  were  satisfied 
with  our  defined  measures. 

6.3.3  Define  Sampling  Procedure 

Defining  the  validation  experiment  eventually  requires  that  we  select  a  set  of  components  and 
assemblies  for  measurement.  This  can  be  viewed  as  selecting  components  and  assemblies 
from  a  population,  with  one  important  proviso:  if  we  expect  a  PECT  to  work  for  all  future 
assemblies,  we  must  assume  that  not  all  of  its  components  and  assemblies  exist.  So  a  question 
arises  as  to  how  to  select  from  a  population  of  nonexistent  entities.  The  three  main  approaches 
to  selecting  samples  from  a  population  are  random,  convenience,  and  judgment  sampling. 

Random  sampling  is  useful  when  samples  can  truly  be  selected  randomly.  Convenience  sam¬ 
pling  occurs  when  the  sample  is  taken  while  considering  some  a  priori  knowledge.  Judgment 
sampling  occurs  when  the  sample  is  selected  explicitly  and  sufficient  domain  knowledge 
exists  to  judge  a  sample  set  that  represents  the  whole  population.  Of  the  three  methods,  ran¬ 
dom  sampling  provides  the  best  statistical  data,  but  it  might  be  difficult  to  obtain  a  real  random 
set  of  samples.  On  the  other  hand,  convenience  and  judgment  sampling  are  more  practical  and 
easier  to  perform,  but  the  resulting  statistical  data  might  be  biased  and  therefore  not  fully  rep¬ 
resentative. 

For  ^abA’  random  sampling  was  used  to  collect  latency  measures  from  a  set  of  synthetic  com¬ 
ponents — components  whose  functionality  consists  of  consuming  CPU  resources.  We  used  a 
combination  of  judgment  and  random  sampling  to  collect  measures  for  assembly  latency  mea¬ 
sures,  and  we  used  judgment  sampling  to  define  variation  points  for  assemblies.  Doing  so  cre¬ 
ated  an  assembly  design  space  for  constructing  viable  assemblies.  Then,  from  that  design 
space,  we  selected  assemblies  randomly.  For  example,  in  the  controller  PECT,  no  realistic 
assemblies  have  more  than  50  components.  Thus,  that  was  an  upper  bound  for  one  variation 
point  in  the  assembly  design  space  that  we  defined. 

In  an  analytic  study,  we  must  define  the  design  space  and  select  a  random  sample  from  it.  Our 
approach  to  validating  A, aba  ^o  exploit  the  “pure  composition”  aspects  of  Pin  to  define 

various  dimensions  of  variation— the  number  of  input  and  output  connectors,  the  number  of 
instances  of  each  type  of  component,  and  so  forth — and  thereby  sketch  the  outlines  of  the 
design  space.  Using  these  variations,  we  generated  the  pseudorandom  assemblies. 

We  have  made  only  tentative  steps  in  understanding  how  to  analytically  characterize  the 
design  space  and  draw  a  random  sample  from  it.  At  this  point,  we  have  only  two  conjectures. 
The  first  is  that  a  component  model  that  provides  well-defined  and  restricted  rules  for  allow¬ 
able  patterns  of  component  interaction  is  likely  to  be  better  suited  for  statistical  analysis  than  a 
wholly  unconstrained  component  model.  The  second  conjecture  is  that  product  line  settings 
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may  augment  a  component  model’s  structure-oriented  rules  with  rules  governing  semantic 
variation  (i.e..  product  feature  variation  [Clements  02a]). 

Section  B.2.3  on  page  98  describes  the  details  of  the  sampling  procedure  for  both  components 
and  assemblies  used  in  the  empirical  validation  of  the  controller  PECT. 


6.3.4  Develop  Measurement  Infrastructure 

As  with  any  experimental  process,  a  measurement  infrastructure  must  be  developed  to  collect 
data.  Where  objective  measurements  are  taken,  as  with  infrastructure  must  be  suffi¬ 

ciently  well  constructed  to  produce  reliable  and  repeatable  measurements.  For  ^aba* 
developed  a  shared-memory  mechanism  for  capturing  traces  of  pin  activation  and  deactiva¬ 
tion.  We  also  developed  a  distributed,  trace  mechanism  based  on  the  Universal  Datagram  Pro¬ 
tocol  (UDP)  for  measuring  higher  order  SAS  assemblies  (i.e.,  an  assembly  of  operator  and 
controller  subassemblies). 

The  measurement  infrastructure  must  also  be  sufficiently  well  documented  to  allow  the  results 
to  be  repeated,  possibly  by  disinterested  third  parties  not  associated  with  the  original  valida¬ 
tion  experiment.  Just  what  should  be  documented  has  much  to  do  with  the  property  of  interest. 
Often,  it  is  clear  which  environmental  factors  have  a  direct  correlation  to  a  property,  for 
instance,  latency.  Latency  for  a  component  will  vary  from  environment  to  environment,  based 
on  processor  speed.  Other  environmental  characteristics  could  have  an  indirect  correlation  to  a 
property — which  may  not  be  immediately  obvious — ^for  example,  the  compiler  (or  compiler 
flags)  used  to  produce  the  component. 

Describing  the  test  environment  becomes  increasingly  important  when  trying  to  establish 
causal  relationships  between  runtime  and  observed  behavior  and  is  especially  critical  when  a 
discrepancy  exists  between  the  predictions  and  observations.  Trying  to  find  a  correlation 
between  such  environmental  characteristics  and  behaviors  of  components  and  assemblies  can 
be  difficult.  Altering  the  environment  to  limit  or  prevent  unexpected  behaviors  is  one  viable 
approach.  However,  having  a  description  about  what  characteristics  were  present  during  the 
validation  may  hold  clues  as  to  what  was  and  was  not  considered  to  be  possible  sources  of 
error. 

For  ^aba’  *he  environmental  characteristics  that  were  recorded  for  the  validation  included 
platform  hardware  characteristics  (e.g.,  processor,  bus,  and  memory  speed),  software  charac¬ 
teristics  (e.g.,  operating  system  and  compiler  versions),  and  residual  processes  (e.g.,  those 
background  tasks  running  at  the  time  of  the  validation).  These  characteristics  are  discussed  in 
Appendix  B.  A  summary  of  the  overall  measurement  and  analysis  infrastructure  developed  for 
X-aba  discussed  in  Appendix  B. 
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6.3.5  Collect  Validation  Data 


This  step  more  or  less  speaks  for  itself.  The  only  issue  that  requires  attention  is  that  a  realistic 
validation  experiment  will  generate  a  large  volume  of  data  and  may  require  significant  com¬ 
puting  resources  and  time  to  do  so.  All  generated  data  should  be  archived  so  that  it  can  be  sub¬ 
jected  to  historical  analysis. 


6.3.6  Analyze  Results 

As  noted  earlier,  the  primary  objective  of  empirical  validation  is  to  assign  statistical  labels  to 
components  and  property  theories.  The  computation  of  these  labels  should  be  straightforward, 
given  an  adequate  definition  of  goals,  measures,  and  sampling  procedures.  However,  this  does 
not  mean  that  the  only  utility  of  empirical  validation  is  the  assignment  of  labels.  In  particular, 
the  data  generated  by  the  validation  experiment  can  be  an  invaluable  resource  for  improving 
the  quality  of  the  property  theory. 

In  the  case  of  X^i^bA'  evaluation  goals  were  not  satisfied  when  the  MRE  was  computed  for 
the  full  sample  of  predictions,  as  reported  in  Table  3  on  page  46.  Instead  of  an  upper  bound  of 
5%  MRE,  we  had  achieved  an  upper  bound  of  only  10%  (albeit  with  much  higher  confidence 
than  required).  As  discussed  in  detail  in  Appendix  B,  analysis  of  the  data  revealed  that  the  pre¬ 
dictions  were  partitioned  into  two  main  regions:  ^table  and  accurate  predictions  (<5%  MRE) 
and  stable  inaccurate  predictions  (the  complement).  From  our  experience  in  co-refinement  and 
earlier  spot  validations,  systematic  error  is  less  problematic  than  random  error  precisely 
because  it  is  repeatable  error.  The  more  optimistic  result  shown  in  Table  4  can  be  anticipated 
under  the  assumption  that  the  source  of  systematic  error  can  be  identified  (as  others  had  been 
during  the  course  of  co-refinement). 


Table  4:  (Optimistic)  Distribution-Free  Tolerance  Interval  for  X aba 


Part  of  Interval 

Description 

N  =  47  (removed  systematic  error) 

sample  size 

Y  =  0.9907 

confidence  level 

p  =  0.80 

population 

MmRE  =  0.01 

MRE 

UB  =  4% 

upper  bound 

Although  the  results  in  the  above  table  are  hypothetical,  this  hypothesis,  combined  with  the 
raw  data  generated  by  the  validation  process,  would  provide  useful  input  to  a  second  iteration 
of  co-refinement  if  it’s  needed. 
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7  Safety  and  Liveness  Analysis  in  the 
Controller  PECT 


This  report  is  focused  on  an  empirical  property  theory  (^aba)  i^s  validation.  Equally  vital 
to  PECT  are  formal  property  theories.  In  this  chapter,  we  illustrate  the  main  ideas  of  one 
approach  to  formally  verifying  safety  and  liveness  properties  of  components  and  assemblies — 
model  checking.  In  that  approach,  a  system  model  is  extracted  from  component  reactions  and 
their  compositions  in  an  assembly.  Safety  and  liveness  conditions  are  expressed  in  a  modal 
(temporal)  logic  and  tested  by  an  exhaustive  search  of  the  state  space  described  by  the  system 
model. 

We  describe  only  the  modeling  and  verification  of  temporal  claims  against  a  single  component 
in  the  controller  assembly.  We  defer  a  description  of  composed  controller  models  and  verifica¬ 
tions  to  a  follow-on  report.  A  formal  description  of  composition  in  Pin  is  provided  by  Ivers 
and  associates  [Ivers  02]. 


7.1  Model  Checking 

Model  checking  is  a  formal  method  for  verifying  finite-state  concurrent  systems.  It  is  based  on 
algorithms  for  system  verification  that  operate  on  a  system  model  usually  expressed  in  some 
form  of  state-transition  representation. 

This  system  model  is  then  checked  against  a  set  of  desired  properties  expressed  in  a  modal 
(temporal)  logic.  Model  checking  is  a  desirable  verification  approach,  because  it  is  automated 
and  exhaustive.  State-transition  representations,  in  essence  finite  state  machines,  are  intuitive 
to  the  average  engineer.  Temporal  logics,  such  as  computational  tree  logic  (CTL)  and  linear 
temporal  logic  (LTL),  are  not  as  intuitive,  but  can  be  mastered  with  practice  [Prior  62],  [Pnueli 
85].  Verification  of  the  system  is  accomplished  by  “checking”  the  model — exhaustively 
searching  the  state  space  described  by  the  system  model  and,  for  each  state,  checking  whether 
the  desired  properties  have  been  satisfied. 
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Characteristics  of  system  models  that  favor  model  checking  over  other  behavioral  verification 
techniques  include 

•  ongoing  I/O  behavior  (in  so-called  “reactive”  systems) 

•  concurrency  (not  a  single  flow  of  control) 

•  control  intensive  (minimal  data  manipulation) 

The  controller  assembly  has  each  of  the  above  characteristics,  indicating  that  model  checking 
is  likely  to  be  a  good  fit. 

The  next  section  addresses  how  we  apply  model  checking  in  the  context  of  a  PECT. 


7.2  Process 

The  goal  of  our  analysis  is  to  determine  whether  the  assembly  of  components  used  to  imple¬ 
ment  the  controller  satisfies  various  safety  and  liveness  properties.  Therefore,  we  begin  by 
producing  a  model  of  the  software  components  used  to  implement  the  controller.  In  this  exam¬ 
ple,  we  focus  on  the  CSWI  component  of  the  SEI  switch  controller  (see  Figure  9  on  page  23). 

First,  we  build  a  Pin  model  of  the  CSWI  component,  basing  the  model  on  the  implemented 
component,  rather  than  its  requirements.  This  is  an  important  distinction;  we  care  about  what 
the  implemented  component  actually  does,  not  what  it  should  do.  Essentially,  we  want  to 
ensure  a  strong  correspondence  between  the  model  and  the  component  implementation. 
Extraction  of  the  model  from  the  implementation  is  one  way  to  accomplish  this,  even  when 
performed  manually,  as  in  this  example.  (Alternatively,  we  could  have  built  the  model  and 
generated  the  implementation;  however,  since  we  already  had  the  latter,  that  approach  was  not 
useful.) 

Next,  we  use  a  model  checker  to  analyze  the  CSWI  model.  However,  the  Pin  model  uses  a  for¬ 
mal  language — CSP  [Hoare  85] — that  is  not  understood  by  the  model  checker  used  in  this 
exercise — ^NuSMV  [Cimatti  00].  Consequently,  we  must  define  an  interpretation  of  Pin  to 
NuSMV.  In  this  exercise,  the  interpretation  is  performed  manually,  although  there  is  no  obvi¬ 
ous  reason  why  it  cannot  be  automated. 

Once  the  CSWI  model  is  expressed  in  a  form  that  is  understood  by  the  model  checker,  we  for¬ 
malize  the  safety  and  liveness  properties  we  want  to  analyze  and  use  the  model  checker  to 
determine  whether  the  model  satisfies  them.  Any  failure  that  is  uncovered  by  the  tool  must 
then  be  analyzed.  Most  failures  include  a  counterexample — an  execution  trace  in  which  the 
desired  property  is  not  satisfied. 
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A  failure  could  mean  one  of  three  things: 


1.  There  is  an  error  in  the  CSWI  component  model.  If  the  counterexample  is  not  a  possible 
trace  of  the  implemented  component,  the  formal  component  model  must  be  corrected. 

2.  There  is  an  error  in  the  formalization  of  the  safety  or  liveness  property.  If  the  counterex¬ 
ample  does  not  violate  the  desired  property  we  meant  to  check,  the  property  has  not  been 
formalized  correctly. 

3.  The  model  does  not  satisfy  the  property.  If  the  counterexample  is  a  possible  trace  of  the 
implemented  component  and  fails  to  satisfy  the  desired  property,  we  have  a  real  error,  and 
the  component  implementation  must  be  repaired. 

Any  type  of  failure  can  mean  that  we  iterate  back  through  the  steps  of  this  process.  Modeling 
and  analysis  are  often  iterative  processes  that  conclude  only  when  a  statement  can  be  made 
regarding  whether  the  component  satisfies  its  required  safety  and  liveness  properties. 

The  steps  of  this  process  are  elaborated  in  the  following  sections. 


7.3  Building  the  Model 

We  assume  that  a  Pin  model  of  an  assembly  is  available,  such  as  the  one  shown  in  Figure  9  on 
page  23.  As  noted,  the  model  checker  we  used,  NuSMV,  does  not  accept  input  in  the  formal 
notation  used  by  Pin.  Consequently,  we  translate  our  Pin  model  into  a  formal  notation  that  the 
model  checker  will  accept.  The  steps  to  translating  the  model  include  (corresponding  to  Sec¬ 
tions  7.3. 1-7. 3. 4,  respectively) 

1 .  Determine  the  scope  of  problem  analysis. 

2.  Produce  a  Pin  model  of  the  problem. 

3.  Translate  the  Pin  model  into  a  state  machine. 

4.  Translate  the  state  machine  into  NuSMV. 

These  steps  are  elaborated  in  the  following  sections. 


7.3.1  Determining  the  Scope  of  Problem  Analysis 

The  context  diagram  shown  in  Figure  16  shows  the  controller  assembly  in  Pin  and  highlights 
the  region  of  that  assembly  used  for  our  verification.  The  CSWI  component  contains  the  logic 
for  interfacing  to  the  operator  and  carrying  out  the  selection  and  operation  protocol  with  the 
XCBR  component.  The  XCBR  component  controls  the  signals  sent  to  the  physical  switch,  and 
contains  protection  logic  (via  PIOC)  and  the  logic  needed  to  accept  commands  (i.e.,  select, 
open,  and  close)  from  the  operator  (via  CSWI). 
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Figure  16:  Controller  System  Diagram 

As  shown  in  Figure  16,  the  operator  has  two  inputs  to  the  controller — opsel  and  oppos — that 
indicate  requests  to  select  or  position  the  switch.  The  CSWI  component  can  send  two  outputs 
to  the  XCBR  component — sbosel  and  sbopos — the  effect  of  which  is  to  select  or  position  the 
switch.  Before  continuing,  CSWI  always  waits  until  it  receives  a  reply  from  XCBR  indicating 
that  the  task  has  been  performed.  XCBR  has  two  outputs  to  the  physical  switch  where  a  selec¬ 
tion  or  position  change  actually  occurs — swsel  and  swpos.  Two  inputs  from  the  physical 
switch — sosel  and  sopos — provide  acknowledgement  feedback  to  XCBR,  through  the  respec¬ 
tive  sbo  channels,  and  ultimately  to  the  operator  (perhaps  in  the  form  of  indicator  lights,  etc.). 


In  this  discussion,  we  focus  on  modeling  the  behavior  of  the  controller  assembly’s  CSWI  com¬ 
ponent. 


7.3.2  Producing  a  Pin  Model  of  the  Problem 

CSWI  has  two  sink  pins — opsel  and  oppos — each  of  which  is  a  synchronous  mutex  pin.  Addi¬ 
tionally,  each  sink  pin  is  threaded,  and  the  two  sink  pins  are  handled  by  the  same  thread.  CSWI 
also  has  two  synchronous  source  pins — sbosel  and  sbopos. 
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Because  CSWI  only  has  one  thread  of  control,  it  is  formally  modeled  in  CSP  as  a  single  reac¬ 
tion.  This  specification  is  shown  in  Figure  17.  Note  that  the  leading  underscore  character  (_)  in 
event  names  indicates  the  initiation  of  an  interaction  on  a  pin  of  that  name;  the  absence  of  a 
leading  underscore  indicates  the  completion  of  an  interaction. 


CSWI  =  _opsel.on  ->  _sbosel ! on  ->  sbosel  ->  opsel  ->  Selected 
[]  _opsel.off  ->  _sbosel!off  ->  sbosel  ->  opsel  ->  CWSI 
Selected  =  _opsel.on  ->  _sbosel!on  ->  sbosel  ->  opsel  -> 
Selected 

[]  _opsel.off  ->  _sbosel!off  ->  sbosel  ->  opsel  ->  CSWI 
[]  _oppos?x  ->  _sbopos!x  ->  sbopos  ->  _sbosel!off  -> 
sbosel  ->  oppos  ->  CSWI 


Figure  17:  CSWI  Pin  Specification 


7.3.3  Translating  the  Pin  Modei  into  a  State  Machine 

While  the  Pin  model  of  CSWI  is  based  on  CSP,  NuSMV  (our  target  model  checker)  requires 
input  in  the  form  of  a  finite  state  machine.  As  shown  in  Figure  18,  we  translated  the  Pin  model 
to  a  simple  state  machine  as  an  intermediate  representation. 

The  transitions  in  that  diagram  correspond  to  CSP  events  in  the  Pin  model.  The  names  associ¬ 
ated  with  these  transitions,  and  subsequently  in  the  NuSMV  model,  are  based  on  the  names  of 
CSP  events  in  the  Pin  model.  For  example,  the  Pin  model  defines  the  operator  selection  event 
as  _opsel,  with  a  data  parameter  of  either  on  or  off,  written  as  _opsel .  on  or 
_opsel .  off,  respectively.  To  preserve  traceability  to  event  names  in  the  Pin  model,  the 
state  machine  modifies  this  convention  slightly  by  changing  _opsel .  on  to  _opsel_on  or 
_opsel .  off  to  _opsel_of  f .  This  transformation  is  necessitated  by  the  syntax  of 
NuSMV,  which  uses  the  period  (.)  operator  for  a  different  purpose.  Consequently,  periods  (as 
well  as  exclamation  points  [!]  and  question  marks  [?])  were  changed  to  underscores. 
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start 


^opseLon/ 

„sbosel_on 


sbosei/ 

opsel 


_oppos_open/ 

_sbopos_open 


Figure  18:  CSWI  State  Machine 


Seven  states  compose  CSWI:  waiting,  selecting,  selected,  opening,  closing,  deselecting,  and 
unselecting. 


The  states  and  major  transitions  are  described  below. 


•  waiting  state:  CSWI  starts  in  the  waiting  state  and  transitions  to  another  state  only  when  an 
_opsel_on  or  _opsel_of  f  event  occurs.  An  _opsel_on  event,  for  example,  indi¬ 
cates  that  the  operator  is  selecting  a  switch  and  results  in  a  transition  to  the  selecting  state. 
As  part  of  this  transition,  CSWI  initiates  an  _sbosel_on  event.  The  “event  1 /even t2” 
convention  indicates  that  an  occurrence  of  eventi  not  only  triggers  a  transition,  but  also 
results  in  the  generation  of  event2.^'*  This  convention  is  how  we  reflect  the  behavior  found 
in  the  Pin  reaction  specification. 

•  selecting  state:  CSWI  remains  in  the  selecting  state  until  it  receives  an  acknowledgement 
that  the  selection  has  been  accomplished,  an  sbosei  output  is  received,  and  an  opsel  input 
is  generated.  CSWI  then  transitions  to  the  selected  state. 

•  selected  state:  CSWI  waits  in  the  selected  state  until  one  of  the  following  events  occurs: 

An  _opsel_of  f  event  is  received,  indicating  the  operator  action  of  deselecting  the 
switch  and  causing  a  transition  to  the  unselecting  state. 

An  _oppos_open  event  is  received,  indicating  the  operator  action  of  requesting  to 
open  the  switch  and  causing  a  transition  to  the  opening  state. 


24.  This  follows  the  convention  of  Harel’s  statecharts,  which  are  reflected  in  UML  [Rumba ugh  CX)],  where  *‘event2’*  is  called  “ac¬ 
tion." 
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An  _oppos_close  event  is  received,  indicating  the  operator  action  of  requesting  to 
close  the  switch  and  causing  a  transition  to  the  closing  state. 

An  _opsel_on  event  is  received,  indicating  the  operator  action  of  selecting  the 
switch  and  causing  a  transition  back  to  the  selecting  state. 

•  opening  state:  If  CSWI  is  in  the  opening  state  and  an  sbopos  output  is  received,  CSWI 
transitions  to  the  deselecting  state  and  generates  an  _sbosel_of  f  event  to  deselect  the 
switch,  since  the  positioning  is  complete.  If  the  acknowledge  is  not  received,  the  system 
continues  to  wait  in  the  opening  state. 

•  closing  state:  If  CSWI  is  in  the  closing  state  and  an  sbopos  output  is  received,  CSWI  tran¬ 
sitions  to  the  deselecting  state  and  generates  an  _sbosel_of  f  event  to  deselect  the 
switch,  since  the  positioning  is  complete.  If  the  acknowledge  is  not  received,  the  system 
continues  to  wait  in  the  closing  state. 

•  deselecting  state:  If  CSWI  is  in  the  deselecting  state  and  an  sbosel  output  is  received, 
CSWI  transitions  to  the  waiting  state  and  generates  an  oppos  input. 

•  unselecting  state:  If  CSWI  is  in  the  unselecting  state  and  an  sbosel  output  is  received, 
CSWI  transitions  to  the  waiting  state  and  generates  an  opsel  input. 

Although  they  are  similar,  the  deselecting  and  unselecting  states  are  both  needed;  the  select 
and  operate  actions  require  different  acknowledgements  to  be  generated,  even  though  the  tran¬ 
sition  to  each  is  caused  by  the  same  thing — an  sbosel  output. 

7.3.4  Translating  the  State  Machine  into  NuSMV 

This  section  discusses  the  translation  of  the  model  from  the  state  machine  represented  in  Fig¬ 
ure  18  to  the  NuSMV  textual  model  shown  in  Appendix  C.  It  was  done  in  a  straightforward 
manner  following  the  steps  listed  below.  Note  that  since  the  Pin  model  indicates  that  CSWI 
has  a  single  thread  of  control,  there  is  no  need  to  use  multiple  processes  in  the  NuSMV  model 

1.  The  transitions  in  the  state  machine  (e.g.,  _opsel__on  and  _oppos_open)  appear  in 
the  NuSMV  model  as  symbolic  names  associated  with  the  input  variable,  which  can  be 
assigned  any  of  the  input  values  shown  in  the  state  diagram.  The  input  variable  can  also 
have  the  value  “none,”  indicating  that  no  inputs  currently  exist.  The  input  variable  is 
declared  as  an  IVAR,^^  directing  NuSMV  to  consider  all  combinations  of  input  sequences 
when  analyzing  claims. 


25.  IVAR  is  a  keyword  from  the  NuSMV  tool.  It’s  short  for  “input  variable.” 


CMU/SEI-2002-TR-031 


59 


2.  The  outputs  in  the  state  machine  (e.g.,  _sbosel_on  and  _sbopos_open)  appear  in 
the  NuSMV  model  as  symbolic  names  associated  with  the  output  variable,  which  is 
declared  as  a  VAR."*^  As  with  input,  output  can  have  the  value  “none,”  indicating  that 
no  outputs  currently  exist.  The  output  variable  is  changed  only  within  the  NuSMV 
model.  The  other  VAR  declaration  is  the  state  variable,  which  takes  on  the  state  names 
from  the  state  machine  (e.g.,  waiting  and  selecting). 

3.  There  is  a  case  expression  relating  the  behavior  of  the  output  variable  at  the  next  state  to 
the  current  values  of  the  state  and  input  variables.  For  example,  the  first  rule  states 
that  if  the  current  state  is  waiting  and  the  current  input  is  _opsel_on,  the  next 
value  of  the  output  is  _sbosel_on.  Hence,  each  output  is  delayed  by  a  single  tick 
from  the  condition  causing  it.  The  last  rule  in  the  case  expression  (1 :  none)  indicates 
that  if  none  of  the  other  rules  in  the  case  expression  apply  (e.g.,  if  the  state  is  waiting 
and  the  input  is  _oppos_open),  the  output  variable  will  default  to  “none.” 

4.  The  other  case  expression  relates  the  changes  in  the  state  variable  to  the  current  state 
and  the  input  variable.  For  example,  if  the  state  is  waiting  and  the  input  is 
_opsel_on,  the  next  state  will  become  selecting.  The  last  rule  (1 :  state)  indicates 
that  if  none  of  the  other  rules  apply,  the  next  state  remains  equal  to  the  current  state. 


7.4  Analyzing  the  Model 

Once  we  have  a  model  of  CSWI  in  a  form  that  NuSMV  understands,  we  can  apply  model 
checking  to  determine  whether  the  model  (and,  therefore,  the  component’s  implementation) 
satisfies  various  system  properties.  For  example,  this  type  of  analysis  can  determine  whether 
deadlock  is  possible. 

In  this  section,  we  begin  by  discussing  different  types  of  system  properties  that  we  can  analyze 
using  model  checking.  We  then  introduce  temporal  logic,  a  formal  language  commonly  used  to 
express  system  properties.  Finally,  we  show  examples  of  properties  expressed  in  temporal 
logic  that  we  used  in  analyzing  the  CSWI  model. 


7.4.1  Types  of  System  Properties 

The  types  of  system  properties  evaluated  by  model  checkers  are  typically  classified  as  safety 
or  liveness  properties.  Although  other  taxonomies  have  been  proposed  by  Naumovich  and 
Clarke  [Naumovich  00],  we  continue  to  use  the  safety  and  liveness  classification.  Intuitively,  a 
safety  property  specifies  that  “bad  things”  cannot  happen,  and  a  liveness  property  specifies 
that  “good  things”  eventually  happen  [Lamport  77]. 


26.  VAR  is  a  keyword  from  the  NuSMV  tool.  It’s  short  for  “variable." 
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To  illustrate  these  points,  consider  a  software  application  in  which  two  or  more  concurrent 
processes  must  access  the  same  critical  section  of  code.  A  condition  that  must  hold  is  mutual 
exclusion,  which  means  that  two  (or  more)  concurrent  processes  cannot  enter  and  execute  a 
common  critical  section  of  code  simultaneously.  This  condition  can  be  specified  as  the  safety 
property  “two  processes  cannot  be  in  the  same  critical  section  at  the  same  time.”  Another  con¬ 
dition  that  is  usually  desirable  in  concurrent  processes  is  starvation  freedom,  which  means  that 
eventually  each  process  will  enter  the  critical  section.  This  can  be  specified  as  the  liveness 
property  “whenever  process  PI  wants  to  enter  the  critical  section,  it  will  eventually  do  so.” 

7.4.2  Temporal  Logic 

Temporal  logic  is  frequently  used  to  formally  define  safety  and  liveness  properties.  Two  well- 
known  logics  used  in  verification  are  LTL  and  CTL. 

In  LTL,  time  is  viewed  as  a  linear  sequence  of  time  instants,  with  each  time  instant  having 
exactly  one  successor.  The  semantic  intuition  of  LTL  expressions  (where  a  and  b  are  primitive 
prepositions)  is  as  follows: 

-  Da 

This  is  read  as  “henceforth  (or  always)  a”  and  means  that  statement  a  is  true  now  and 
in  all  future  time  instants. 

0  a 

This  is  read  as  “eventually  a”  and  means  that  a  is  true  either  now  or  at  some  time 
instant  in  the  future. 

°a 

This  is  read  as  “in  the  next  state  or  at  the  next  time  instant,  a  will  be  true.” 

-  aUb 

This  is  read  as  “a  is  true  until  b  becomes  true.” 

In  CTL,  time  is  viewed  as  a  branching  tree  of  time  instants,  each  with  one  or  more  possible 
successors.  A  path  is  a  particular  linear  sequence  of  time  instants  (a  path  in  the  tree)  in  which 
each  successive  time  instant  is  one  of  the  possible  successors  of  the  current  time  instant.  The 
semantic  intuition  of  CTL  expressions  (where  p  and  q  are  primitive  prepositions)  is  as  fol¬ 
lows: 


-  AGp 

Along  all  paths,  p  holds  globally  (henceforth). 

-  EGp 

At  least  one  path  exists  where  p  holds  globally. 
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AF  p 

Along  all  paths,  p  holds  at  some  time  instant  in  the  future  (eventually). 

-  EFp 

A  path  exists  where  p  holds  at  some  time  instant  in  the  future. 

-  AXp 

Along  all  paths,  p  holds  in  the  next  time  instant. 

-  A[pUq] 

Along  all  paths,  p  holds  until  q  holds. 

-  E[pUq] 

A  path  exists  where  p  holds  until  q  holds. 

Both  LTL  and  CTL  include  the  usual  a  a  b,  a  v  b,  — la,  a  ^  b  logic  expressions.  While  LTL 
expressions  appear  to  be  a  subset  of  CTL,  their  expressive  power  differs.  It  is  possible  to 
express  properties  in  LTL  that  cannot  be  expressed  in  CTL,  and  vice  versa.  The  NuSMV 
model  checker  allows  claims  to  be  expressed  in  either  CTL  or  LTL;  however,  CTL  was  used  as 
the  primary  specification  language  in  this  experiment. 


7.4.3  CSWi  Claims 

The  desired  system  properties  analyzed  with  respect  to  the  CSWI  model  include  safety  and 
liveness  properties.  The  specific  behavioral  characteristics  of  each  property  are  often  first 
expressed  in  a  natural  English  language  format  and  then  translated  into  CTL  (or  LTL).  These 
temporal  logic  expressions  are  often  called  claims.  Examples  of  claims  for  CSWI  are  dis¬ 
cussed  in  the  following  sections. 

7.4.3.1  Example  Liveness  Claim 

A  number  of  liveness  claims  were  constructed  for  CSWI,  the  most  obvious  of  which  investi¬ 
gate  its  operational  correctness.  Those  claims  follow  the  pattern  of  “if  an  operator  action 
occurs,  a  signal  is  conveyed  to  the  appropriate  entity.”  In  this  example,  the  entity  would  be 
XCBR.  A  specific  desired  liveness  property  is  “if  the  operator  selects  the  switch,  the  ‘select 
before  operate’  signal  is  sent  to  the  XCBR.”  Translating  this  into  the  appropriate  CTL  expres¬ 
sion  results  in 

AG  ((state  =  waiting)  a  (input  =  _opsel_on)  ->  AX(output  =  _sbosel_on)) 

The  above  claim  is  read  as  “for  all  paths  globally,  whenever  CSWI  is  waiting  and  the  operator 
select  comes  on,  the  sbo  select  signal  will  come  on.” 
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7.4.3.2  Example  Safety  Claim 

Safety  properties  are  logical  expressions  of  conditions  that  should  never  occur,  presumably 
because  the  resultant  action  could  endanger  a  person  or  harm  equipment.  One  of  the  opera¬ 
tional  characteristics  of  the  controller  assembly  is  the  enforcement  of  the  “select  before  oper¬ 
ate”  protocol.  This  can  be  rephrased  as  the  desired  safety  property  “under  human  control,  the 
switch  cannot  be  operated  unless  it  is  first  selected.”  Translating  this  into  the  appropriate  CTL 
expression  results  in 

“jE  [->(output  =  _sbosel_on)  U  (output  =  _sbopos_open)] 

The  above  statement  is  read  as  “it  is  impossible  for  the  switch  to  be  opened 
(_sbopos__open)  before  it  has  been  selected  (_sbosel_on).” 

7.4.3.3  Example  Failed  Claim 

The  CSWI  model  does  not  satisfy  the  following  safety  claim: 

AG(input  =  _opseLon  ->  AG  output  =  none) 

This  is  actually  a  good  thing,  because  we  don’t  want  the  safety  claim  to  be  true.  The  claim 
asserts  that  whenever  the  operator  selects  the  switch,  nothing  will  ever  happen  (i.e.,  no  output 
will  be  generated).  When  we  run  this  claim  in  NuSMV,  it  confirms  that  the  model  does  not  sat¬ 
isfy  the  claim,  and  it  also  presents  a  counterexample  showing  a  particular  trace  in  which  the 
claim  is  not  satisfied.  The  counterexample  is  shown  in  Figure  19. 

—  specification  AG  (input  =  _opsel_on  AG  output  =  none) 
is  false 

--  as  demonstrated  by  the  following  execution  sequence 
State  1.1: 
input  =  none 
output  =  none 
state  =  waiting 

State  1.2: 
input  =  _opsel_on 

State  1.3: 

input  =  _oppos_close 
output  =  _sbosel_on 
state  =  selecting 

Figure  19:  NuSMV  Counterexample 
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The  counterexample  represents  three  time  instants  (1.1,  1.2,  and  1.3),  which  have  the  follow¬ 
ing  meanings: 

•  1.1;  CSWI  begins  in  the  initial  state,  waiting,  and  there  is  no  input  or  output. 

•  1.2:  An  input  of  _opsel_on  is  received,  indicating  that  the  operator  selects  the  switch. 

•  1.3:  An  output  of  _sbosel_on  is  generated,  and  CSWI  transitions  to  the  selecting  state. 

In  the  final  time  instant,  1.3,  CSWI  generates  an  output,  violating  the  claim  that  nothing  will 
happen  after  an  operator  selects  the  switch. 

The  NuSMV  model  and  the  claims  verified  against  it  are  contained  in  Appendix  C. 
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8  Discussion 


The  basic  structure  and  concepts  of  a  PECT  had  been  prototyped  before  the  SAS  model  prob¬ 
lems  were  defined  [Hissam  02].  Nonetheless,  the  SAS  problems,  although  simple  by  the  stan¬ 
dards  of  production  systems,  were  complex  enough  to  reveal  previously  unappreciated,  or 
underappreciated,  subtleties  of  PECT.  We  discuss  some  of  those  subtleties,  with  an  emphasis 
on  those  aspects  of  PECT  that  were  shown  to  require  further  study.  In  Section  8.1,  we  discuss 
methodological  results,  while  in  Section  8.2  we  briefly  touch  on  the  model  solutions. 


8.1  Results  on  Method 

8.1.1  interfaces  Between  Specialized  Skills 

One  striking  observation  is  that  designing,  implementing,  and  validating  a  PECT  requires  a 
combination  of  specialized  expertise.  Some  indication  of  this  can  be  gleaned  from  the  different 
workers  identified  in  Chapter  4.  Table  5  focuses  not  on  workers,  but  on  their  skills,  and  in  par¬ 
ticular  those  skills  needed  to  build  the  SAS  PECT.  To  some  extent,  these  skills  are  coherent 
with  the  workers  identified  earlier.  However,  it  is  significant  that  we  often  found  it  necessary 
for  a  single  worker  to  straddle  several  skills,  suggesting,  as  is  anticipated  in  the  RUP,  that  a 
single  person  will  fill  many  roles,  perhaps  simultaneously.  Note  that  some  divergence  from  the 
general  set  of  workers  and  their  implied  skills  outlined  in  Chapter  4  is  expected,  given  our 
focus  on  research  rather  than  production. 


Table  5:  Areas  of  Expertise  Needed  to  Build  SAS  Operator  and  Controller 

PECTs 


Area  of  Expertise 

Use  of  Expertise  in  SAS  Model  Solutions 

component  technology 

Design  the  Pin  component  model,  application  program  interface  (API), 
runtime  environment,  and  deployment  model. 

component-based 

development 

Implement  the  component  technology  and  define  and  implement  con¬ 
formant  SAS  components  (including  their  behavioral  models)  and  their 
SAS  assemblies. 

systems  programming 

Implement  real-time  extensions  to  Windows  2000,  observation  mecha¬ 
nisms  for  validation  environments,  and  low-level  features  of  the  compo¬ 
nent  runtime  environment. 

measurement,  valida¬ 
tion,  experimental  design, 
statistical  analysis 

Define  measurement  scheme  and  statistical  analysis  and  labeling 
approach  for  component  execution  time  and  latency  property  theories. 
Conduct  component  measurement  and  empirical  validation. 
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Table  5:  Areas  of  Expertise  Needed  to  Build  SAS  Operator  and  Controller 
PECTs  (Confd.) 


Area  of  Expertise 

Use  of  Expertise  in  SAS  Model  Solutions 

performance  modeling 

Define  the  latency  property  theory  based  on  the  theory  underlying 

RMA. 

model  checking 

Develop  model-checker-specific  state  models  and  define,  develop,  and 
check  temporal  claims  of  controller  safety  and  liveness  conditions. 

language  design  and  for¬ 
mal  semantics 

Design  the  specification  language(s)  for  Pin  assemblies  and  the  formal 
(CSP)  compositional  semantics  of  connectors  between  component 
(CSP)  behaviors. 

electronics 

Design,  develop,  and  calibrate  test  hardware  to  simulate  switch  control. 
This  custom  hardware  was  used  to  empirically  validate  the  controller 
latency  theory. 

substation  automation 

Define  the  model  problems  and  evaluation  criteria  such  that  the  PECT 
reflects  a  representative  class  of  design  and  engineering  problems. 

As  discussed  in  Chapter  5,  co-refinement  is  a  means  by  which  actors  possessing  expertise  in 
two  domains,  each  with  specialized  conceptual  schemes  and  vocabularies,  can  nonetheless 
negotiate  mutual  satisfaction  of  a  shared  concern  (practicability),  while  ensuring  satisfaction 
of  their  unique  concerns  (generality  and  tractability).  Co-refinement  is  a  negotiation  process 
that  establishes  a  shared  conceptual  interface  between  those  actors.  Note  that  although  the 
interface  is  conceptual,  it  serves  an  analogous  role  to  the  more  familiar  notion  of  software 
interface:  it  helps  to  make  explicit,  and  therefore  manage,  the  dependencies  in  a  system. 


Skills  Interface: 
Co-Refinement 


U  _ _ _ _ _ _ _ _  _  _  J 


Figure  20:  Interfaces  Among  PECT  Development  Skills 

This  metaphor  of  negotiation  is  apt  and  can  be  applied  elsewhere  in  the  PECT  development 
process.  Consider,  for  example,  the  interface  between  the  skills  depicted  in  Figure  20.  Above 
or  beneath  each  skill,  we  placed,  in  brackets  ([]),  a  one-word  descriptor  for  the  PECT  design 
concern  that  motivates  the  application  of  that  skill.  The  associations  are  labeled  with  a  one-  or 
two-word  descriptor  of  a  shared  concern.  In  Figure  20,  we  show  a  shared  concern  between 
component  technology  and  measurement  validation — “certifiable.”  This  concern  covers  issues 
such  as  whether  the  measured  property  can  be  validated  by  an  independent  party,  whether  the 
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measures  fall  within  required  confidence  or  tolerance  intervals,  or  whether  the  level  of  system¬ 
atic  and  random  measurement  error  can  be  quantified.  Measurement  and  validation  also  share 
a  concern  with  the  domain  specialist — whether  the  assemblies  used  for  empirical  validation 
are  representative  of  real  design  problems. 

It  is  unreasonable  to  expect  a  single  person  to  possess  the  range  of  expertise  reflected  in  Figure 
20.  A  transitionable  PECT  development  method  must  provide  a  clear  distribution  of  these 
forms  of  expertise  across  different  workers  and  processes  that  define  a  separation  of  the  skills 
across  activities. 

8.1 .2  Infrastructure  Complexity 

Carl  Sagan  introduced  his  television  series,  Cosmos,  with  the  statement,  “To  bake  an  apple  pie, 
we  must  first  create  the  universe." 

Before  we  can  assemble  components  and  predict  their  aggregate  properties,  we  must  create  a 
PECT  infrastructure  that  includes  the  collection  of  programs  exclusive  o/ application-level 
(e.g.,  controller-level)  components  and  their  assemblies.  The  view  of  PECT  depicted  in  Figure 
5  on  page  13,  The  Four  Environments  of  a  PECT,  is  a  good  guide  to  what  constitutes  infra¬ 
structure.  Nonetheless,  that  structure  is  guilty  of  oversimplification.  The  practicality  of  PECT 
rests,  to  some  extent,  on  the  cost  of  developing  and  maintaining  its  infrastructure.  We  must 
therefore  strive  to  understand  which  part  of  a  PECT  infrastructure’s  complexity  is  inherently 
related  to  PECT,  and  which  part  arises  from  concerns  that  have  already  been  addressed  by 
available  technology  or  is  inherent  to  a  problem  domain  such  as  SAS. 

One  approach  to  obtaining  this  understanding  is  to  examine  Figure  5  from  the  perspective  of  a 
software  development  environment  (SDE).  Clearly,  more  is  required  of  an  SDE  than  the 
assembly  and  analysis  environments  shown  in  Figure  5.  An  SDE  would  also  include  facilities 
for  configuration  and  version  management,  and  a  repository  of  components  and  assemblies, 
for  example.  The  components  themselves  will  likely  be  implemented  in  a  conventional  pro- 
grarhming  language  such  as  C#,  C-H+,  or  Java.  This  suggests  that  code  browsing,  compiling, 
linking,  and  debugging  facilities  are  also  required  in  a  PECT.  Still,  none  of  this  is  part  of  a 
PECT-specific  infrastructure,  and  it  is  all  readily  available — for  example,  Microsoft’s  Visual- 
Studio  and  Borland  C-H-’s  Developer.  What  remains  is  to  ensure  that  a  PECT  infrastructure  is 
integrated  with  an  existing  SDE.  While  not  trivial,  this  is  not  a  substantial  risk  to  PECT. 

Another  way  to  look  at  Figure  5,  however,  is  from  the  perspective  of  a  language-specific  SDE 
(LS-SDE).  A  language  comprises  its  syntax  and  semantics.  In  PECT,  the  syntax  of  the  lan¬ 
guage  is  defined  by  the  constructive  model  [Ivers  02].  However,  the  language  used  to  specify 
(constructive)  assemblies  for  any  PECT  is  not  necessarily  simple  in  structure;  its  syntax  and 
semantics  are  dependent  on  an  open-ended  set  of  environment  types  and  analysis  views,  only 
some  of  which  may  be  applicable  in  any  given  application.  This  added  complexity  must  be 
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laid  directly  on  the  PECT  doorstep — it  cannot  be  waived  off  as  a  simple  matter  of  SDE  tool¬ 
ing. 

Some  of  the  infrastructure  needed  to  support  such  a  semantically  extensible  LS-SDE  can  be 
regarded  as  a  one-time  investment — once  developed,  a  constructive  model  and  its  infrastruc¬ 
ture  can  be  used  repeatedly  for  a  family  of  applications  [van  Ommering  02].  This  suggests  that 
a  PECT  may  be  more  easily  justified  in  the  context  of  a  product  line.  Although  product  lines 
require  a  substantial  organizational  commitment,  they  are  justified  by  business  considerations 
[Clements  02a],  and  PECT  may  in  fact  make  product  lines  more  effective.  On  the  other  hand, 
there  is  not  likely  to  be  an  infinite  supply  of  useful  constructive  models.  For  example.  Pin 
bears  a  strong  resemblance  to  other  component  languages,  notably  Darwin  [Magee  93], 
Wright  [Allen  97],  and  the  emerging  UML  V2.0  standard.^^  There  is  hope,  then,  for  conver¬ 
gence  on  an  industry-standard  component  language. 


8.1 .3  Design  Space:  Rules  for  Inclusion  and  Exclusion 

We  have  adopted  the  idea  of  a  PECT  design  space  as  a  basis  for  validating  empirical  property 
theories — in  this  report,  latency.  The  term  design  space  has  a  metaphorical  connotation,  but  it 
is  given  concrete  meaning  in  empirical  validation  through  its  definition  as  a  set  of  discrete 
variation  points.  That  is,  we  define  design  space  as  an  N-dimensional  cartesian  coordinate  sys¬ 
tem, where  coordinates  on  each  dimension  take  values  from  a  (small)  finite  set  representing 
one  possible  variation  at  that  variation  point,  or  on  that  dimension.  From  this  design  space,  we 
select  the  sample  of  assemblies  to  use  for  the  statistical  analysis  of  predicted  versus  observed 
assembly  behaviors. 

The  coordinate  system  for  this  design  space  defines  the  set  of  assemblies  that  are  included 
within  the  scope  of  a  property  theory.  That  is,  the  coordinate  system  defines  inclusion  rules  for 
assemblies.  That  definition  reflects  a  positive  statement  of  where  a  property  theory  must  be 
valid,  in  the  sense  that  any  sample  taken  from  this  design  space  will  be  representative  of  the 
class  of  systems  to  which  this  PECT  will  be  applied.  We  want  the  predictions  to  be  valid  for  all 
members  of  this  class.  There  is  evidently  a  close  connection  between  this  concept  of  design 
space  and  that  of  variation  points  in  product  line  architecture  (e.g.,  Theil  and  Hein’s  work 
[Theil  02]),  and  in  fact  we  use  the  term  variation  point  to  emphasize  this  connection. 

In  practice,  however,  we  must  also  define  rules  for  exclusion  from  the  design  space.  That  is, 
we  will  almost  certainly  want  to  characterize  not  just  where  we  think  the  property  theory 


27.  UML  V2.0  is  currently  a  work  in  progress.  See  <http://www,uml,org/umI/>  for  more  information  about  it. 

28.  To  be  more  precise,  the  coordinate  system  under  discussion  defines  sufficient  conditions  for  the  assembiies  of  a  class  of 
products;  the  necessary  conditions  are  defined  by  the  constructive  modei  and  its  interpretation  under  the  property  theory 
being  validated.  For  example,  the  interpretation  of  the  controller  latency  theory  X^ba  restricts  assembly  topologies  to  only 
those  that  satisfy  the  priority  ceiling  constraint.  Ali  assembiies  must  satisfy  these  purely  syntactic,  and  therefore  necessary, 
conditions. 
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should  be  valid,  but  also  where  we  know  it  is  not  valid.  One  way  to  approach  this  is  to  discover 
the  analysis  limits  for  each  variation  point.  The  analysis  limits  of  a  variation  point  define  the 
greatest  lower  bounds  and/or  the  least  upper  bound  for  the  set  of  values  defined  for  each  varia- 
tion  point."  More  simply,  each  variation  point  can  be  tested  to  destruction  to  find  its  analysis 
limits.  This  is  analogous  to  the  approach  taken  by  Gorton  and  Liu  [Gorton  02];  however,  their 
measurement  object  is  a  large-scale  commercial  off-the-shelf  (COTS)  component  (an  Enter¬ 
prise  JavaBeans  server)  and  not  the  property  theory  used  to  predict  performance. 

In  general,  there  has  been  insufficient  work  in  software  engineering  research  regarding  the 
empirical  validation  of  design  theories.  PECT  provides  a  conceptual  vocabulary  and  technical 
means  for  making  much  needed  progress  in  that  area. 

8.1.4  Confidence  Intervals  for  Formal  Theories 

If  there  has  been  insufficient  work  in  the  empirical  validation  of  design  theories  based  on 
observation  (and,  therefore,  measurement),  there  has  been  almost  no  work  in  the  empirical  val¬ 
idation  of  theories  based  on  formal  logic.  (Barnett  and  Schulte’s  work  is  an  exception  [Barnett 
01].)  Indeed,  the  very  notions  seem  to  be  mutually  exclusive,  which  may  account  for  the  lack 
of  attention  to  this  subject.  Nonetheless,  assertions  about  safety  and  liveness  properties  of 
assemblies  (as  discussed  in  Chapter  7)  are  made  with  respect  to  a  model  of  the  components’ 
behaviors  and  their  interactions.  We  can  have  100%  confidence  that  a  claim  is  satisfied  or  fal¬ 
sified  by  a  model.  However,  there  are  two  more  substantial  questions: 

1.  Can  we  have  objective  confidence  that  the  model  corresponds  to  reality? 

2.  Can  we  have  confidence  that  the  model  is  sufficient  to  justify  the  claims? 

The  answer  to  question  1  depends,  to  some  extent,  on  how  the  correspondence  (the  hypothe¬ 
sized  “satisfies”  relation)  between  the  model  and  reality  is  established  in  the  first  place.  It 
might  be  established  through  automated  (implying  formal)  translation.  Two  forms  of  auto¬ 
mated  translation  are  possible:  generating  components  from  models  (e.g.,  Sharygina’s  work 
[Sharygina  02])  and  generating  models  from  components  (e.g.,  Havelund  and  Pressburger’s 
work  [Havelund  00]).  The  first  form  is  common  in  practice,  while  the  second  is  more  of  a 
research  topic.  In  both  cases,  our  confidence  in  the  satisfaction  claim  is  proportional,  if  not 
equal,  to  our  confidence  in  the  correctness  of  the  translator  implementation.  This  does  not  lead 
to  infinite  regress,  however,  since  we  have  an  empirical  means  of  establishing  this  confidence 
through  the  testing  of  the  translator.  If,  however,  the  correspondence  between  the  model  and 
component  is  manual,  there  is  no  basis  for  objective  confidence  in  the  hypothesized  “satisfies” 


29.  We  assume  that  the  set  of  values  for  each  variation  point  is  partially  ordered.  This  assumption  seems  reasonable  if  there  is 
a  correspondence  between  the  variation  point  and  some  property  theory  parameter.  Only  those  variation  points  will  be  of 
interest  when  testing  that  theory’s  failure. 
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relation.  The  implication  is  that  we  must  use  automation  to  generate  models  and/or  compo¬ 
nents  in  our  approach  to  PECT. 

The  answer  to  question  2  is  more  subtle  and  rests  on  equivalence  relations  over  models.  Ask¬ 
ing  whether  a  model  satisfies  an  implementation  (we  now  use  this  term  in  place  of  reality)  is 
just  a  more  specific  form  of  the  question  of  whether  two  models  are  equivalent  for  some  pur¬ 
pose.  Sometimes,  but  by  no  means  always,  the  more  concrete  model  is  said  to  refine,  or  be  a 
refinement  of,  the  more  abstract  model;  the  equivalence  relation  is  then  called  refinement.  A 
property  proven  to  hold  for  a  model  will  also  hold  for  its  refinements.^®  However,  there  are 
many  different  ways  of  defining  refinement.  Roscoe  defines  a  hierarchy  of  three  forms  of 
refinement  for  the  CSP  process  algebra — trace,  trace  failures,  and  failures-divergence  refine¬ 
ment  [Roscoe  98];  Davies  (citing  work  by  Reed  [Reed  88])  describes  a  lattice  of  nine  refine¬ 
ment  equivalences  for  CSP  [Davies  93].  Not  to  be  outdone,  van  Glabbeek  [van  Glabbeek  01] 
identifies  13  equivalence  relations,  not  all  based  on  CSP  traces.  In  each  case,  different  notions 
of  equivalence  can  be  used  to  justify  different  claims. 

We  can  draw  two  inferences  from  the  discussion  of  question  2.  First,  we  should  leave  the  defi¬ 
nition  of  equivalence  relations  over  formal  models — and  the  theorem  provers  based  in  them — 
to  the  experts;  we  should  apply  the  fruits  of  their  efforts  in  PECT.  Second,  a  PECT  infrastruc¬ 
ture  must  be  flexible  enough  to  accommodate  a  variety  of  formal  theories,  each  of  which  pro¬ 
vides  a  notion  of  semantic  equivalence  sufficient  to  make  the  kinds  of  specification  claims 
required. 


8.2  Results  on  Model  Solutions 

Three  model  problems  were  posed,  one  leading  to  an  operator-level  PECT,  one  to  a  controller- 
level  PECT,  and  one  to  a  higher  order  PECT  for  assemblies  of  assemblies.  In  the  final  analysis, 
only  the  operator  and  controller  PECTs  were  developed.  While  a  higher  order  operator/con¬ 
troller  assembly  was  demonstrated,  no  property  theory  was  developed  for  higher  order  assem¬ 
blies. 

The  operator-level  PECT  was  developed  without  incident.  However,  it  became  apparent  dur¬ 
ing  its  development  that  the  operator  interface  was  quite  simple — in  fact,  only  reentrant  sink 
pins  were  required.  Moreover,  operator  display  latency  is  completely  dominated  in  any  higher 
order  assembly — the  time  required  to  interact  with  the  display  and  perform  data  validation  on 
operator  input  is  minor  (often  less  than  a  simple  scheduling  question),  compared  with  network 
and  controller  latency. 


30.  This  is  actually  too  strong  a  statement;  not  all  properties  are  preserved  under  refinement — deadlock  freedom,  for  example. 
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Nonetheless,  we  developed  a  simple  latency  property  theory  for  the  operator  PECT  that 
proved  to  be,  under  the  highly  constrained  constructive  model  used,  highly  accurate.  A  sum¬ 
mary  of  the  property  theory’s  performance  is  shown  in  Figure  21.  We  do  not  summarize  the 
results  of  this  performance  as  a  confidence  or  tolerance  interval,  since  the  simplicity  of  the 
PECT  led  us  to  focus  our  validation  efforts  elsewhere. 


Predicted  vs.  Measured 


- -  Predicted  start  ->  end 

■X - ^ —  C1  to  End,  after 

-■ - ■ —  Cl  to  Cl .  after 


Figure  21:  Results  of  Spot  Validation  of  Operator  PECT  Latency  Theory 

The  controller-level  PECT  presented  considerably  more  methodological  and  technological 
challenges,  most  of  which  have  been  discussed  already.  To  reiterate  the  main  results,  the 
produces  stable  predictions.  The  population  as  a  whole  has  an  MRE  ~4%,  which,  in  the  aggre¬ 
gate,  leads  to  an  upper-bound  99%  confidence  interval  in  an  upper  bound  of  10%  MRE.  The 
latter  does  not  meet  our  objective  of  a  5%  upper  bound.  On  the  other  hand,  such  stable  results 
are  still  quite  good  on  a  non-real-time  platform  such  as  Windows  2000.  And,  on  the  good 
assumption  that,  with  reasonable  effort,  stable  errors  can  be  eliminated  (where  the  word 
“error”  means  >5%MRE,  currently  exhibited  by  -25%  of  the  population — see  Figure  57  on 
page  121),  we  achieve  an  upper  bound  of  4%  MRE  while  maintaining  99%  confidence. 

The  validation  exercise  was  more  interesting  for  its  methodological  implications  than 
for  its  validation  of  a  specific  latency  model.  The  initial  validation  data  revealed  a  significant 
quotient  of  systematic  and  random  error.  Five  separate  validation  runs  were  required  to  stabi¬ 
lize  the  model.  The  first  three  served  to  remove  errors  in  the  measurement  infrastructure 
itself — the  instrumentation  was  developed  in  tandem  with  the  validation  work  and  was  there¬ 
fore  itself  a  work  in  progress.  The  last  two  runs  served  to  remove  inconsistencies  between  the 
model  and  the  controller  runtime  implementation.  In  other  words,  the  empirical  validation 
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exposed  an  inconsistency  between  model  assumption  and  runtime  implementation;  moreover, 
the  raw  data  produced  by  the  validation  allowed  us  to  quickly  identify  the  sources  of  error. 

It  is  interesting  to  speculate  on  one  similarity  between  the  development  and  use  of  empirical 
and  formal  models  that  this  validation  experience  (see  Appendix  B  on  page  93)  demonstrates; 
both  processes  are  highly  iterative.  The  nature  of  iteration  with  empirical  models  is  discussed 
in  Chapter  5  and  in  the  validation  exercise.  These  experiences  mirror  the  experiences  of  engi¬ 
neers  who  have  developed  models  for  temporal-logic  model  checking.  In  that  case,  iteration  is 
required  to  develop  a  model  that  can  be  model  checked  and  that  exposes  the  right  level  of 
details  to  satisfy  a  temporal  claim,  and  to  formulate  the  correct  claims  in  the  first  place. 


72 


CMU/SEI-2002-TR-031 


9  Next  Steps 


As  noted  in  the  introduction  to  this  report,  the  objective  for  this  work  was  exploration — the 
technical  concepts  of  PECT  and  the  methods  for  developing  and  validating  a  PECT.  As  with 
any  exploration,  a  body  of  knowledge  was  generated  that  must  now  be  consolidated.  Improved 
understanding  leads  to  new,  and  more  challenging,  questions  that  must  be  explored. 

In  this  chapter,  we  summarize  the  most  important  areas  of  consolidation  and  exploration. 
Motivating  these  priorities  is  our  plan  to  extend  the  work  reported  here  during  the  next  year  to 
accommodate  more  rigorous  feasibility  requirements  and,  working  with  our  industry  partner, 
ABB,  to  incorporate  feasibility  requirements  drawn  from  the  domains  of  substation  automa¬ 
tion  and  industrial  robotics. 


9.1  PECT  Infrastructure 

The  infrastructure  of  a  PECT  is  only  a  means  to  an  end — predictable  assembly  from  certifiable 
components.  However,  it  is  a  necessary  means,  and  our  ability  to  “scale  up”  the  PECT 
approach  requires  some  up-front  investment  in  that  infrastructure.  Some  of  this  investment  has 
already  occurred  as  a  by-product  of  producing  SAS  model  solutions  and  their  PECTs;  more 
needs  to  be  done. 


9.1 .1  Measurement  and  Validation  Environment 

A  substantial  suite  of  tools  was  developed  to  measure  component  execution  time,  automati¬ 
cally  generate  assembly  samples,  interpret  those  assemblies  to  instantiate  and  operate  on  anal¬ 
ysis  views,  capture  traces  of  assembly  execution,  and  perform  statistical  analysis  of  the  timed 
execution  traces  vice  their  predicted  traces.  Most  of  the  tools  in  this  tool  suite  were  used  for 
both  the  operator  and  controller  PECTs,  suggesting  that  those  tools  may  well  form  the  core  of 
a  common  measurement  and  validation  environment  for  future  PECT  development. 

The  future  work  required  is  consolidative  in  nature;  it  involves  integrating,  refining,  and  rug- 
gedizing  the  existing  tool  set.  Integration  will  achieve  the  end-to-end  automation  that  is 
required  to  generate  the  large  data  sets  necessary  to  empirically  validate  a  substantial  design 
space.  This  automation  will  require  more  attention  to  the  interfaces  defined  between  tools  and 
to  data  interchange  syntax  and  semantics.  A  good  foundation  for  data  interchange  has  been 
established  by  using  XML  to  externalize  component  assemblies,  but  this  extemalization  must 
be  made  consistent  with  an  evolving  constructive  model  and  its  specification  language. 
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9.1.2  Core  Pin  Language  and  Assembly  Environment 

One  of  the  tenants  of  the  PECT  approach  is  that  complexity  can  be  “packaged"  and  hidden 
from  end  consumers.  Still,  there  will  be  some  minimum,  irreducible  complexity  exposed  by 
each  analysis  view.  If  this  complexity  is  to  be  manageable  in  the  aggregate,  we  must  strive  to 
eliminate  as  many  forms  of  unnecessary  complexity  as  possible.  To  (mis)appropriate  a  now 
famous  political  aphorism,  '‘it’s  the  packaging,  stupid!” 

The  process  used  in  the  SAS  model  solutions  for  creating  and  deploying  assemblies  of  compo¬ 
nents  was  tedious  and  error  prone.  Component  properties  and  assemblies  of  components  were 
encoded  directly  in  XML,  and  although  the  connector  “wiring”  was  generated  automatically 
from  XML,  there  is  no  disguising  the  awkwardness  of  representation  and  process.  A  visual 
composition  environment  (e.g.,  see  the  work  of  Plakosh  and  associates — specifically  Chapter 
5  [Plakosh  99])  is  a  form  of  packaging  that  can  dramatically  simplify,  and  make  more  pleasur¬ 
able,  the  process  of  attributing  component  behavior,  and  developing,  analyzing,  and  deploying 
their  assemblies. 

As  discussed  in  Section  8.1.2  on  page  67,  a  (visual)  PECT  assembly  environment  is  a  form  of 
LS-SDE — in  this  case,  one  that  emphasizes  pure  composition.  The  future  work  needed  in  this 
area  is  a  mix  of  exploration  and  consolidation,  and  requires  a  definition  of  a  core  language  for 
the  Pin  component  model  (see  the  work  of  Ivers  and  associates  for  a  start  at  this  [I vers  02])  and 
the  development  of  a  visual  environment  to  support  it  (a  small  matter  of  programming).  By 
core,  we  mean  a  graphical  and  textual  syntax  (including  a  syntax  for  reaction  rules)  and  syn¬ 
tax-directed  extemalization. 


9.1 .3  Real-Time  Component  Runtime  Environment 

One  pleasant  surprise  in  developing  the  SAS  model  solutions  was  that  the  component  runtime 
environments  for  the  operator  and  controller  were  rather  simple  to  develop.  Only  the  controller 
environment  posed  a  challenge  due  to  the  artificial  circumstance  of  our  building  a  priority- 
based  real-time  PECT  on  an  operating  system  (Windows  2000)  with  very  limited  support  for 
priority-based  scheduling.  Even  so,  the  operator  and  controller  component  runtime  environ¬ 
ments  emerged  largely  from  a  restriction  of  component  freedom  to  only  a  limited  set  of  envi¬ 
ronment  (runtime)  interfaces. 

Still,  for  the  next  stage  of  PECT  development,  more  care  must  be  taken  when  selecting  an 
underlying  operating  system  and  specifying  the  environment  types  for  hierarchical  assemblies 
(see  Section  3.3.2  on  page  17) — the  latter  supports  distributed  assemblies.  A  definition  of 
these  environment  types  will  also  be  essential  to  the  compositional  semantics  of  environment- 
specific  connectors  added  to  the  core  Pin  component  model. 
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9.1.4  Model  Checking  (Analysis)  Environments 

The  SAS  model  problems  were  focused  primarily  on  theories  of  latency.  For  PECT  to  be  suc¬ 
cessful,  it  must  be  applicable  to  other  behavioral  theories.  Compositional  theories  of  reliability 
for  components  and  assemblies  are  also  of  interest  to  software  industry.  Empirical  theories  for 
compositional  reliability  are  possible,  but  generalized,  falsifiable  theories  have  proven  to  be 
elusive.  See  the  work  of  Mason  [Mason  02]  and  Stafford  and  McGregor  [Stafford  02]  for  an 
entree  to  this  topic.  Formal  proofs  of  correctness  have  traditionally  been  thought  of  only  as  a 
means  of  obtaining  reliability,  not  for  predicting  it.  However,  as  discussed  in  Section  8.1.4  on 
page  69,  there  may  be  an  empirical  dimension  to  formal  theories  of  correctness  after  all. 

Complete  proofs  of  correctness  remain  problematic,  but  within  recent  years,  substantial 
progress  has  been  made  in  improving  the  practicality  of  technologies  for  so-called  partial 
proofs  of  correctness.  In  particular,  proofs  based  on  the  exhaustive  model  checking  of  specific 
safety  and  liveness  properties  specified  in  a  temporal  logic  are  becoming  almost  mainstream. 
For  recent  developments  in  model  checking  of  control  systems,  see  Sharygina’s  work  [Shary- 
gina  02],  for  details  on  process  algebra  and  model  checking  of  concurrent  Java  programs,  see 
the  work  of  Magee  and  Kramer  [Magee  99],  and  for  a  more  theoretical  perspective  on  the 
model  checking,  see  the  work  of  Clarke  and  associates  [Clarke  99].  These  results  and  those 
related  to  them  are  finding  their  way  into  practice.  For  example,  Microsoft  is  using  composi¬ 
tional  verification  to  certify  third-party  device  drivers  [Ball  02].^*  Our  principle  motivation  for 
using  the  CSP  process  algebra  to  specify  models  of  component  behavior  (i.e.,  reactions)  is  that 
models  of  assembly  behavior  can  be  composed  for  analysis  automatically. 

Our  concrete  next  step  is  to  develop  back  ends  to  generate  lower  level  model  representations 
for  use  in  formal  model  checkers  such  as  SMV^^  [Cimatti  00],  Failures-Divergence  Refine¬ 
ment  [Gardiner  00],  and  SPIN  [Holzman  97],  probably  in  that  order.  Although  this  is  not  triv¬ 
ial,  it  is  fairly  simple.  More  complexity  is  involved  in  the  question  of  compositional  reduction 
and  abstraction,  and  other  techniques  to  minimize  the  state  space  of  composed  assembly 
behaviors.  Our  approach  is  to  defer  as  many  of  these  details  to  the  back-end  analysis  tool  as  is 
practical  and  to  introduce  such  techniques  in  the  Pin  infrastructure  only  as  needed. 


9.2  PECT  Method 

This  report  has  outlined  in  a  broad  way  the  processes  involved  in  developing  and  validating  a 
PECT.  The  general  area  of  certification  is  ripe  for  consolidation  into  a  PECT  validation  guide 
and  further  exploration  into  the  certification  locale  (i.e.,  the  environment  where  the  certifica¬ 
tion  will  be  done).  The  next  steps  in  these  areas  are  outlined  below. 


31 .  For  more  information,  see  <http://research.microsoft.com/slam/>. 

32.  SMV  is  a  symbolic  model-checking  tool  developed  at  Carnegie  Mellon  University.  For  more  information  about  it,  see 
<http://www.cs.cmu.edu/-modelcheck/smv/smvmanual.ps>. 
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9.2.1  PECT  Empirical  Validation  Guide 

How  is  a  validation  experiment  constructed?  How  are  measurements  collected,  and  how  is 
systematic  and  random  error  quantified?  How  is  systematic  error  removed?  And  how  is  the 
remaining  error  propagated?  How  many  samples  are  required  to  achieve  the  desired  statistical 
confidence?  Which  form  of  confidence  interval  is  most  appropriate?  How  is  the  assembly  pop¬ 
ulation  defined,  and  how  is  the  sample  (or  frame)  of  that  population  selected?  What  inferences 
can  be  drawn  given  this  sampling  procedure?  What  statistical  tools  can  be  used  to  analyze 
unexpected  results? 

Most  of  these  questions  are  reflected  in  the  workflows  discussed  for  empirical  validation  in 
Chapter  6  and  illustrated  in  detail  in  Appendix  B.  However,  our  workflows  are  descriptive 
rather  than  prescriptive — they  describe  what  steps  should  be  performed,  but  do  not,  in  most 
cases,  specify  how.  Any  prescriptions  provided  are  quite  general.  As  a  result,  the  skills  needed 
to  validate  a  PECT  are  not  well  contained  within  the  current  method;  a  more  prescriptive  guide 
for  laboratory  validation  work  is  a  necessary  consolidation  effort.  The  objective  of  a  prescrip¬ 
tive  guide  would  be  to  describe  the  validation  process  and  identify  (if  not  describe)  the  experi¬ 
mental  and  statistical  foundations  necessary  for  conducting  a  sound  validation.  Elements  of 
such  a  guide  are  outlined  by  Moreno  and  associates  [Moreno  02],  but  many  more  elements  are 
needed. 

9.2.2  Certification  Locale  and  Foundations  for  Trust 

Our  work  so  far  has  focused  on  establishing  component  measures  in  a  laboratory  setting.  We 
have  not  yet  addressed  issues  of  how  these  measures  can  be  established  independently.  Of 
equal  importance  to  the  ability  of  third  parties  to  validate  the  predictive  power  of  a  PECT  is 
their  ability  to  assign  trusted  and  objective  properties  to  components.  One  aspect  is  the  poten¬ 
tial  of  insurance  underwriting  (e.g.,  see  the  work  of  Li  and  associates  [Li  02])  as  a  basis  for  a 
weaker  notion  of  trust,  which  we  may  call  confidence.  This  is,  however,  just  one  aspect  of  a  set 
of  aspects  required  to  establish  what  amounts  to  a  social  network  of  trust.  We  plan  to  explore 
them  in  the  context  of  our  next  round  of  industry  model  problems. 
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Appendix  A  X^ba  Property  Theory 


^ABA  allows  US  to  predict  the  average  latency  of  concurrent  periodic  tasks  on  a  single  proces¬ 
sor.  When  applied,  via  interpretation,  to  Pin  assemblies,  X^gA  can  be  used  to  predict  the  aver¬ 
age  latency  of  arbitrary  execution  paths  through  an  assembly. 

Property  theories  are  based  on  models.  A  model  is  a  simplified  representation  of  a  real  system 
that  captures  those  system  aspects  that  are  relevant  for  gaining  insight  about  some  system 
property  or  behavior.  Models  can  be  analytic,  simulation  based,  or  a  hybrid  of  both.  An  ana¬ 
lytic  model  is  based  on  mathematical  relations  among  variables  and  intended  to  be  solved 
mathematically.  A  simulation  model  is  a  computer  program  that  reproduces,  to  some  level  of 
detail,  the  behavior  of  a  real  system.  Simulation  models  are  generally  used  instead  of  analytic 
models  when  it  is  impossible,  or  very  difficult,  to  create  an  analytic  model. 

^ABA  is  a  hybrid  analytic-simulation  model.  The  essential  concepts  of  are  described  in 
Section  A.2,  and  the  simulator  algorithms  are  discussed  in  Section  A.  3.  Motivating  aspects  of 
the  Pin-X^gA  interpretation  are  also  discussed  in  Section  A.3.  Section  A.4  documents  the  syn¬ 
tax-directed  translation  scheme  used  to  implement  that  interpretation. 


A.1  Essential  >.aba  Concepts 

Because  the  periods  of  tasks  need  not  be  the  same,  the  latency  of  any  task,  sometimes  referred 
to  as  a  job,  may  be  different  due  to  the  phasing  tasks  and  their  priorities.  This  effect  is  shown 
in  Figure  22;  in  particular,  notice  that  the  start  of  task  T2  can  be  delayed  by  higher  priority 
tasks  and  that  the  execution  time  (i.e.,  start  time  to  finish  time)  for  those  tasks  can  vary  from 
two  to  three  units  of  time. 
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Task  execution  -  The  start  of  execution  is  not  delayed  and  the  task  is  not  preempted. 


Task  execution  -  The  start  of  execution  is  delayed  by  a  higher  priority  task(s),  but  the  task  is  not  preempted  during  execution. 

Task  execution  -  The  start  of  execution  is  not  delayed,  but  the  task's  execution  is  preempted  by  a  higher  priority  lask(s)  during 
execution. 

Task  execution  -  The  start  of  execution  is  delayed  by  a  higher  priority  lask(s),  and  execution  is  preempted  by  a  higher  priority 
task(s)  during  execution. 


Figure  22:  Task  Phasing  and  Hyper-Period 

Given  N  tasks  7),  we  define  the  hyper-period^^  (HP)  as  the  least  common  multiple  (LCM)  of 
the  periods  of  all  tasks.  As  shown  in  Figure  22,  the  hyper-period  is  the  amount  of  time  until  the 
pattern  of  execution  for  a  set  of  tasks  repeats.  We  only  need  to  analyze  a  hyper-period,  because 
it  covers  all  the  possible  latencies  that  a  job/task  will  exhibit.  After  a  hyper-period,  the  pattern 
of  latencies  repeats,  meaning  that,  for  each  task,  the  number  of  latency  predictions  can  be  com¬ 
puted  as  follows: 


NP 


HP 

Tj. period 


A  task  Tj  has  a  property,  Tj.offset,  that  indicates  the  arrival  time  of  the  task’s  first  job.  Each 
task  Tj  consists  of  a  sequence  of  subtasks,  Tj.subtasks,  that  perform  the  actual  task’s  computa¬ 
tion.  Each  subtask  Sj  has  a  priority  Sj.priority  and  an  execution  time  Sj.C.  These  concepts  and 
their  relations  are  depicted  in  Figure  23. 


33.  Sometimes  also  referred  to  as  cycle  time  in  real-time  literature. 
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AnalyticAssembly 


K>- 


tasks 


1. 


Task 

subtasks 

• — 

Subtask 

period:  Natural 
offset:  Natural 

▼ 

C:  real 

priority:  Natural 

Figure  23:  Elements  of  the  Analytic  Model 

The  following  are  the  major  assumptions  of  X^ba- 


•  A  task  is  the  only  unit  of  concurrency. 

•  Each  component  has  a  unique  priority. 

•  A  task  can  be  preempted  only  by  a  higher  priority  task. 

•  If  no  higher  priority  task  is  ready,  a  running  task  runs  up  to  completion. 

•  Tasks  do  not  yield  the  CPU  until  they  complete  a  job. 

•  Tasks  do  not  block  on  I/O. 

^ABA  *^0^5  not  address  the  situation  where  tasks  can  be  blocked  waiting  on  an  I/O  request,  but 
it  can  handle  blocking  on  a  semaphore  to  enter  a  critical  section.  If  two  or  more  tasks  share  a 
critical  section,  that  section  is  modeled  as  a  subtask  in  each  task.  In  Figure  24,  the  shared  criti¬ 
cal  section  is  subtask  C.  Task  A  consists  of  subtasks  Aj,  A2,  C,  A4,  and  A5;  task  B  consists  of 
subtasks  Bj,  C,  and  B3. 


Task  A 

_ A 

SBHBI 

1  r 

M4  I 

> - 

B, 

TaskB 

Figure  24:  Blocking  Example 

The  priority  of  critical  section  C  must  be  assigned  according  to  the  priority  ceiling  protocol. 
That  is,  its  priority  has  to  be  higher  than  the  highest  priority  of  all  the  subtasks  immediately 
preceding  the  critical  section.  For  example,  if  the  priority  of  A2  is  2  and  the  priority  of  B ,  is  4, 
the  priority  of  C  has  to  be  at  least  5. 


A.2  Simulator 

As  explained  in  Chapter  5,  the  initial  X*  property  theories  were  purely  analytic.  When  we 
wanted  to  know  not  only  how  the  latency  of  a  task  would  be  affected  by  blocking,  but  also 
when  blocking  would  occur,  the  amount  of  information  in  the  model  made  the  formulation  of  a 
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purely  analytic  model  quite  awkward.  We  developed  a  simulation  to  keep  track  of  the  state  of 
all  the  tasks  while  advancing  time. 

Several  analytic  computations  are  used  by  the  simulator,  thus  making  a  hybrid  analytic- 
simulation  theory.  For  example,  the  end  of  the  hyper-period  is  computed  as 


HPEnd  =  maxj-i  t^jCTj.offset)  +  LCMj-i  NfTj. period) 


Also,  the  arrival  time  of  the  next  task  at  or  after  time  t  for  task  Tj  is  computed  as 


ai(t) 


t-Tj. offset 
Tj. period 


(Tj. period)  +  Tj. offset 


The  simulation  is  event  driven  (commonly  referred  to  as  discrete-event  simulation)  and  con¬ 
trolled  by  a  main  loop  that  advances  the  time  from  one  scheduling  event  to  the  next.  A  sched¬ 
uling  event  is  a  point  in  time  when  a  task  must  be  stopped,  started,  or  resumed. 

In  each  iteration  of  the  simulation,  the  main  loop  creates  a  list  with  the  next  scheduling  event 
for  each  task;  sorts  it  by  time,  priority,  and  type;  and  then  selects  the  one  to  be  handled.  Instead 
of  the  main  loop  maintaining  a  list  of  pending  events,  each  task  knows  what  it  has  to  do  next; 
start,  run,  or  stop.  Tasks  are  told  to  advance  their  internal  clock  to  the  time  of  the  event  being 
handled,  and  only  one  of  them  has  the  CPU  while  doing  this  (i.e.,  the  task  assumes  it  executed 
while  advancing  the  clock).  Figure  25  shows  the  pseudocode  for  the  prediction  main  loop. 
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predict ()  { 

Initialize  sleep  queue  to  empty 
Initialize  run  queue  to  empty 

Compute  the  hyper-period 

Initialize  all  tasks  and  place  them  on  the  sleep  queue 

Set  t  =  to  earliest  task  start  time  on  the  sleep  queue 

while  t  <  end  of  first  steady-state  hyper-period  { 

Place  all  tasks  on  the  run  queue  from  the  sleep  queue 
whose  starttime  <=  t 

if  there  is  nothing  on  the  run  queue  { 

if  there  is  nothing  on  the  sleep  queue  end  while  loop 
else  set  time  to  next  earliest  task  start  time  on  the 
sleep  queue  and  continue  while  loop 

} 

Take  highest  priority,  longest  waiting  task  from  run  queue 
and  execute  it 

Advance  t  by  execution  time  of  task  just  executed 

if  task  state  after  execution  is  sleeping 
put  it  on  the  sleep  queue 

else 

put  task  back  on  the  run  queue 

} 

} 


Figure  25:  Prediction  Pseudocode 

The  simulator  uses  several  utility  functions  to  initialize,  determine  the  next  scheduling  event, 
advance  the  time,  and  finally,  support  report  generation.  Figures  26  through  28  show  the 
pseudocode  for  the  methods  that  provide  these  services. 


initialize  0  {  . 
let  time  =  0 

let  current  subtask  index  =  first  subtask 
let  priority  =  first  subtask  priority 
let  subtask  time  =  0 
let  task  state  =  Sleeping 

} 


Figure  26:  Initialize  Pseudocode 
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float  ExecuteTask ( )  { 

if  task  is  sleeping  { 

Change  state  to  running 
Record  it's  starttime 

} 

if  task  is  running  { 

let  cpu  time  used  =  subtask  execution  time 

Advance  to  next  subtask 

if  current  subtask  is  last  subtask  { 

//it  has  complete  a  job 
record  job  latency 

let  current  subtask  =  index  to  first  subtask 
let  task  state  =  sleeping 
set  next  execution  start  time 

} 

} 

return  (cpu  time  used) 

} 


Figure  27:  ExecuteTask  Pseudocode 


Vector  getJobLatencies ( )  ( 

return  a  vector  with  the  recorded  latencies  of  all  jobs 

} 


Figure  28:  getJobLatencies  Pseudocode 


A.3  Pin>XABA  Interpretation  (Discussion) 

The  property  theory  models  every  assembly  as  a  set  of  periodic  tasks.  Each  task  has  a 
sequence  of  subtasks,  which  execute  at  specified  priority  levels.  Hence,  the  analytic  interpreta¬ 
tion  consists  in  creating  this  set  of  tasks  from  the  constructive  Pin  assembly.  The  following 
discussion  focuses  on  the  highlights  of  the  interpretation. 


A.3.1  Clock  Components  and  Task  Period 

To  Start,  there  is  no  concept  of  task  and  subtasks  in  Pin.  Given  that  a  task  in  the  analytic  model 
performs  a  computation  periodically,  it  is  intuitive  to  associate  Pin  clock  components  with 
tasks.  A  clock  component  is  provided  by  an  environment;  it  has  no  sink  (i.e.,  it  does  not  exe¬ 
cute  anything)  and  has  a  source  pin  that  is  activated  periodically. 
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A.3.2  Restriction  on  Reactions 


The  Pin  assembly  shown  in  Figure  29  represents  a  task  having  a  period  of  60  milliseconds 
(ms),  with  two  subtasks,  Cj  and  C2.  Here,  we  have  taken  the  liberty  of  representing  a  period  as 
a  parameter  of  a  clock  source  pin.  In  the  future,  this  will  be  represented  using  some  form  of 
annotation  mechanism.  For  the  SAS  prototypes,  these  properties  were  encoded  directly  in  an 
XML  description  of  the  clock  interface. 


Figure  29:  Simple  Constructive  Assembly 

We  impose  restrictions  on  the  form  that  reactions  take  within  components.  In  our  interpreta¬ 
tion,  sink  pins  will  be  mapped  to  tasks.  If  we  allow  components  to  have  the  following  code 
structure  (as  reflected  in  the  reaction  specification  of  components),  we  would  generate  a  task 
timeline  along  the  lines  of  the  one  shown  in  Figure  30. 

Ci.sO  { 

Do  some  computation 
Activate  pin  Ci.r 
Do  some  more  computation 

} 


Figure  30:  Timeline  for  a  Sink  Pin  with  Interactions  in  the  Middle  of  Its  Execution 
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This  would,  however,  violate  our  analysis  assumption  that  a  subtask  has  to  finish  before  its 
successor  can  start.  To  avoid  this  situation,  we  introduce  a  constructive  constraint  requiring 
that  reactions  occur  only  at  the  end  of  a  sink’s  execution.  With  this  constraint,  the  code  for  the 
previous  example  follows  this  pattern: 

Ci.sO  { 

Do  all  some  computation 
Activate  pin  C^.r 

) 

Its  execution  timeline  is  shown  in  Figure  3 1 . 


c2.s 

subtask 
cl.s  I 

f 

i _ 

Figure  31:  Timeline  for  a  Sink  Pin  with  Interactions  at  the  End  of  Its  Execution 

A.3.3  Interpreting  Reentrant  Interactions 

The  sample  assembly  we  have  been  analyzing  so  far  has  a  linear  topology,  making  it  straight¬ 
forward  to  get  the  corresponding  sequence  of  subtasks.  However,  Pin  allows  multiple  source 
pins  to  react  to  one  sink  pin  and  multiple  sink  pins  to  be  coimected  to  one  source  pin,  making 
it  possible — and  perhaps  the  most  common  situation — ^to  have  assemblies  with  tree-like  topol¬ 
ogies.  The  challenge  then  is  to  linearize  that  tree  into  a  sequence  of  subtasks  that  can  be  ana¬ 
lyzed  by  the  property  theory. 

Figure  32  shows  a  Pin  assembly  with  a  topology  that  must  be  flattened  to  make  it  analyzable 
by  the  property  theory.  In  Pin,  the  order  of  interactions  can  be  specified  as  being  deter¬ 
ministic  or  non-deterministic  (as  Ivers  and  associates  discuss  [Ivers  02]).  The  order  of  interac¬ 
tions  can  be  specified  by  annotating  a  Pin  diagram,  but  in  it  and  the  following  assemblies,  we 
consider  the  order  in  Pin  diagrams  to  be  from  top  to  bottom.  So,  in  Figure  32,  Ci.ri'N-»C2.s 
occurs  before  Ci.rj-^C3.Si.  The  reaction  on  Cj.s,  denoted  in  the  annotation  in  Figure  32, 
determines  the  order  of  the  interactions  associated  with  Cj.s.  So,  again  from  Figure  32, 
Cj.ri''-»C2.s  occurs  before  Ci.r2'^C3.S2. 
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Figure  32:  Constructive  Assembly  with  Tree-Like  Topology 

The  assembly,  in  Figure  32,  has  only  one  thread,  which  is  associated  with  Cj.s.  The  C2.s,  C3.SJ, 
and  C3.S2  sinks  execute  in  that  same  thread,  because  they  are  reentrant.  And  because  threads 
are  the  units  of  concurrency  in  Windows,  no  problem  occurs  mapping  the  whole  assembly 
onto  one  analysis  task,  which  is  the  unit  of  concurrency  in  the  analytic  model.  In  this  case,  the 
fact  that  components  Cj.rj'N-»C2.s  and  Ci.rj'N.^C3.S|  are  synchronous  makes  it  easy  to  figure  out 
the  order  in  which  sinks  and,  consequently,  subtasks  will  execute.  The  corresponding  analysis 
task  for  this  assembly  is  <C].s,  C2-s,  C3.S1,  C3.S2>,  where  the  comma  (,)  is  a  sequence  operator, 
and  <a,  b>  is  interpreted  as  a  executes  before  b. 

The  sequence  of  subtasks  can  be  obtained  algorithmically  by  doing  a  pre-order  traversal  of  the 
tree.  Another  algorithm  starts  from  the  leaves  and  works  towards  the  root.  Every  time  we  get 
to  a  node  where  two  branches  are  synchronously  connected,  we  prepend  the  subsequence  for 
the  branch  that  executes  first  to  the  subsequence  for  the  branch  that  executes  second.  For 
example,  we  obtain  the  subsequences  x  =  <C2.s,  C3.Si>  and  y  =  <C3.S2>.  When  we  get  to  Cj.s, 
subsequence  x  is  prepended  to  subsequence  y,  and  then  Cj.s  is  prepended  to  the  result,  obtain¬ 
ing,  as  before,  <Cj.s,  C2.S,  C3.S],  C3.S2>. 


A.3.4  Interpreting  Asynchronous  Interactions 

When  we  introduce  asynchronous  connections  to  the  assembly,  we  also  introduce  more 
threads,  because  asynchronous  sink  pins  have  associated  threads.  The  problem  that  arises  is 
that  now  we  have  many  units  of  concurrency  in  the  constructive  model  (i.e.,  threads),  and  they 
must  somehow  be  mapped  to  tasks  in  the  analytic  model.  The  only  way  to  do  this  is  to  elimi¬ 
nate  real  concurrency  within  the  constructive  assembly.  We  achieve  this  by  assigning  priorities 
to  the  sink  pin  threads  so  that  no  two  sink  pins  have  the  same  priority.  The  subassembly  in  Fig- 
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ure  33  shows  an  example  in  which  we  can  see  how'  concurrency  is  eliminated  from  a  subas¬ 
sembly  with  two  threads  via  priorities. 


Figure  33:  Constructive  Assembly  with  Asynchronous  Connections 

Sink  pins  Cj.s,  C3.S  have  threads;  therefore,  the  top  and  bottom  paths  of  execution  could,  in 
principle,  run  concurrently.  Priorities  allow  us  to  flatten  this  subassembly  into  a  sequence  of 
subtasks. 

The  asynchronous  calls  to  Cj.s  and  C3.S  are  done  before  either  of  the  two  sink  pins  has  a  chance 
to  execute.  Pin  c^.s  will  execute  first,  because  it  has  a  higher  priority  than  Cj.s.  Then,  even 
though  C|.s  is  ready  to  execute,  C4.S  will  execute  because  it  has  a  higher  priority.  When  C4.S 
finishes,  it  activates  C5.S,  but  Cj.s  (which  has  a  higher  priority)  goes  first.  Then  03.8  executes 
and  lastly  C5.S  does.  The  resulting  analysis  task  is  <C3.s,  C4.S,  Cj.s,  03.8,  C5.s>. 

If  we  try  to  interpret  this  assembly  by  doing  a  pre-order  traversal  of  the  tree  as  we  did  before, 
we  will  obtain  the  incorrect  sequence  <Ci.s,  C3.S,  C3.S,  C4.S,  C5.s>.  Nevertheless,  the  right 
sequence  of  subtasks  can  also  be  obtained  algorithmically.  The  sequence  of  subtasks  for  the 
two  synchronous  branches  in  the  assembly  can  be  computed  by  doing  a  pre-order  traversal  as 
we  did  before.  In  that  way,  we  get  the  sequences  x  =  <Cj.s,  C3.s>  and  y  =  <C3.s,  C4.S,  C5.s>. 

When  two  execution  paths  are  initiated  asynchronously,  we  merge  the  two  sequences.  This 
merge  operation  consists  of  creating  a  new  sequence  of  subtasks  by  appending  to  it  the  highest 
priority  subtask  from  the  head  of  the  sequences  being  merged  and  repeating  this  step  until  both 
sequences  have  been  consumed.  Table  6  shows  the  state  of  the  sequences  during  the  merge 
operation  in  the  example. 


86 


CMU/SEI-2002-TR-031 


Table  6:  Merge  Operation  Example 


ClkO.subtasks 

X 

y 

<> 

<C].S,  C2.S> 

<€3.8,  C4.S,  C3.S> 

<C3.S> 

<Ci.S,  C2.S> 

<C4.S,  C5.S> 

<CyS,  C4.S> 

<Cj.8,  C2.S> 

<C5.S> 

<03.5,  C4.S,  C|.S> 

A 

CO 

V 

A 

0 

V* 

V 

<€3.8,  C4.S,  C|.S,  C2-S> 

<> 

<05. 8> 

<€3.8,  C4.S,  Cj.s,  C2.S,  C5.S> 

<> 

<> 

The  merge  operation  requires  only  that  the  two  subtasks  in  the  head  of  the  sequences  being 
merged  do  not  have  the  same  priority.  If  they  did,  the  algorithm  would  not  be  able  to  determine 
which  one  will  execute  first.  It  is  important  to  note  that,  as  long  as  the  previous  condition  is 
met,  different  sink  pins  in  one  task  can  share  the  same  priority  level.  For  the  sake  of  simplicity, 
we  enforced  the  stronger  condition  that  all  threads  in  Pin  have  unique  priority  assignments. 


A.3.5  Combining  Synchronous  and  Asynchronous 
Interactions 

It  may  seem  that  applying  the  prepend  and  merge  operations  on  finding  synchronous  and  asyn¬ 
chronous  connections,  respectively,  will  produce  the  interpretation  of  an  assembly  with  mixed 
synchronous  and  asynchronous  connections.  In  fact,  in  the  example  used  to  present  asynchro¬ 
nous  connections,  there  were  also  synchronous  connections  down  the  branches.  However,  this 
strategy  will  not  render  the  right  analytic  interpretation  if  synchronous  connections  are  fol¬ 
lowed  (from  root  to  leaves)  by  asynchronous  connections,  as  is  the  case  in  the  subassembly 
shown  in  Figure  34. 


Figure  34:  Mixed  Synchronous  and  Asynchronous  Connections 

The  correct  interpretation  of  this  subassembly  is  <C2.s,  Cj.s,  C4.S,  c^.s,  C7.S,  C5.s>.  If  we  work 

from  the  leaves  toward  the  root  applying  the  prepend  and  merge  operations,  we  get  the 
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sequences  x  =  <C2.s,  c^.s,  C7.s>  and  y  =  <C|.s,  C4.S.  C5.s>.  However,  if  we  apply  the  prepend 
operation  as  suggested  for  the  earlier  interpretations  (see  Section  A.3.3),  we  would  get  the 
wrong  sequence  <C2.s,  c^.s.  Cy.s,  C|.s,  C4.S.  C5.s>. 


The  problem  is  that  the  asynchronous  interaction  activates  another  thread,  allowing  more  than 
one  component  to  be  ready  simultaneously.  This  means  that  when  the  two  sequences  are  com¬ 
bined,  the  algorithm  should  determine  which  ready  component  will  execute,  based  on  the 
algorithm’s  priorities.  However,  this  has  to  be  done  after  the  asynchronous  interaction.  This 
operation,  called  combine,  requires  that  when  a  subtask  is  added  to  a  sequence,  whether  it  was 
activated  asynchronously  is  indicated  using  an  overbar  (')over  the  asynchronous  sink  pin’s 
name,  for  example,  <C6.s,  C7.s>.  The  operation  behaves  as  a  prepend  as  long  as  no  asynchro¬ 
nously  activated  subtasks  are  in  the  head  of  either  sequence.  When  one  is,  the  operation 
behaves  as  a  merge  from  that  point  on.  Table  7  shows  the  steps  in  the  combine  operation  for 
the  example. 


Table  7:  Combine  Operation  Example 


CO.subsequence 

SequenceO 

Sequencel 

Mode 

<> 

<C2.S,  Cg.S,  C7.S> 

<Ci.S,  C4.S,  C5.S> 

prepend 

<C2.S> 

<C6.S,  C7.S> 

<Ci.S,  C4.S,  C5.S> 

merge 

<C2.S,  Ci.S> 

<C6.S,  C7.S> 

<€4.8,  C5.S> 

merge 

<C2.S,  Cj.S,  C4.S> 

<C6.S,  C7.S> 

<05.  s> 

merge 

<C2.S,  C].S,  €4,8,  C5.S> 

<  C7.S> 

<C5.S> 

merge 

<C2*S,  C4.S,  C^.S,  C’7.S> 

<> 

<05.  s> 

merge 

<C2.S,  Cj.S,  C4.S,  C5.S,  C7.S,  C5.S> 

<> 

<> 

merge 

A.3.6  Multiple  Asynchronous  Interactions  on  One  Source  Pin 

When  two  sink  pins  are  connected  to  an  asynchronous  source,  the  merge  operation  assumes 
that  they  are  activated  at  the  same  time.  In  practice,  this  is  not  true;  they  are  activated  in  the 
order  specified  in  the  assembly.  A  problem  arises  if  one  of  the  activated  sinks  has  a  priority 
greater  than  that  of  the  activating  source,  because  the  source  will  be  preempted  before  it  fin¬ 
ishes  activating  the  remaining  sinks.  When  that  happens,  the  actual  behavior  may  not  be  what 
the  merge  operation  predicts. 

Figure  35  depicts  an  assembly  with  this  problem.  The  interpretation  just  described  renders  the 
incorrect  sequence  <Ci.s,  C4.S,  C2.S,  Cg.s,  C5.s>  when  the  actual  behavior  is  <C2.s,  Cg.s,  C|.s,  C4.S, 
C5.s>.  The  actual  behavior  differs  because  the  sink  C2-s  preempts  the  thread  on  the  reaction 
(source)  C3.r  before  it  activates  pin  Ci.s. 
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Figure  35:  Multiple  Asynchronous  Connections  Problem 

We  can  remove  this  problem  by  applying  a  rewrite  rule  on  the  constructive  assembly  before 
interpreting  it.  This  preprocessing  consists  of  rewriting  all  the  asynchronous  connections  by 
interposing  a  proxy  component  between  each  source  and  the  asynchronous  connector.  This 
proxy  has  a  reentrant  (and  therefore  unthreaded)  sink  that  is  connected  synchronously  to  the 
source  of  the  original  asynchronous  connection  and  a  source  that  is  connected  to  the  original 
asynchronous  connector.  The  proxy  component  has  no  execution  time,  and  because  it  is 
unthreaded,  it  executes  at  the  same  priority  as  its  caller.  Figure  36  shows  this  rewrite  rule 
applied  to  the  assembly  in  Figure  35. 
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Figure  36:  Multiple  Asynchronous  Connections  Rewrite 

After  preprocessing,  the  interpretation  renders  <pc2.ps,  C2.s,  Cg.s,  pCj.ps,  Cj.s,  C4.S,  C5.s>.  The 
proxy  components  pc^  have  no  execution  time,  so  they  can  be  removed  from  the  sequence. 
Finally,  we  obtain  the  right  execution  sequence  <  C2.S,  Cg.s,  Cj.s,  C4.S,  C5.s>. 


A.4  Pin-^ABA  Interpretation  (Formal) 

The  discussion  so  far  has  been  informal.  To  enable  automated  interpretation,  we  must  com¬ 
plete  the  formalization.  We  chose  to  use  syntax-directed  translation  to  implement  the  interpre- 
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tation  [Aho  87].  This  requires  an  input  grammar  for  the  constructive  assembly  and  a  set  of 
semantic  reductions  that  render  the  interpretation. 

The  grammar  for  constructive  assemblies  is  as  follows: 

asm  — >  src  \  asm  &  src 

src  — >  pin  *  subasm_set  \  pin  +  subasm_set 

pin  — »  component. pinid 

siibasm_set  — >  siibasm  \  ( subasmjist ) 

subasmjist  — >  siibasm  |  subasmjiist ,  subasm 

stibasm  snk  |  snk  -  src_set 

snk  — >  pin 

src_set  — >  5rc  I  (  srcjiist ) 
srcjist  src  \  srcjist ,  src 

where  an  asterisk  (*)  denotes  asynchronous  interaction,  a  plus  sign  (+)  denotes  synchronous 
interaction,  a  dash  (-)  denotes  reaction,  and  a  comma  (,)  separates  source  pins  in  a  reaction  and 
sink  pins  in  a  1:N  interaction.  Note  that  reactions  are  not  expressed  in  CSP,  but  rather  only  in 
the  restricted  form  of  call  dependency. 

Notes: 

•  The  interpretation  assumes  that  all  constructive  constraints  are  honored.  That 
is  why  we  make  no  distinction  in  this  language  for  mutex  sink  pins — their  blocking 
effect  is  addressed  by  the  constructive  constraint  for  the  priority  ceiling. 

•  To  make  the  grammar  simpler,  we  have  disregarded  the  fact  that  the  src  sources  in 
the  first  production  have  to  be  different,  because  their  pins  must  have  an  implicit 
periodic  stimulus.  This  is  the  case  for  the  Clock  component. 

The  syntax-directed  definition  for  obtaining  the  analytic  assembly  for  a  constructive  assembly 
is  shown  in  the  following  table.  Both  synthesized  and  inherited  attributes  are  used.  The 
pseudocode  for  the  types  and  functions  used  in  the  syntax -directed  translation  are  shown  in 
Sections  A.4.7  and  A.4.8. 

To  keep  the  syntax -directed  definition  presented  here  as  simple  as  possible,  some  liberties 
have  been  taken.  The  src  .period  and  pin. priority  attributes  are  used  without  first 
being  assigned  anywhere.  Assigning  them  would  have  required  adding  a  declaration  construct 
to  the  grammar  and  a  symbol  table  to  the  syntax -directed  definition.  Also,  the  src  .period 
attribute  should  have  been  synthesized  from  the  pin  .period  attribute,  which  isn’t 
expressed  in  the  semantic  rules  either. 
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Table  8:  Syntax-Directed  Definition  for  the  Analytic  Interpretation 


Production 

Semantic  Rules 

asm  /E  src 

addtask(src.period,  src.subtasks) 
src.priority  :=  0 

asm  /E  asm  &  src 

addtask(src.period,  src.subtasks) 
src.priority  ;=  0 

src  /E  pin  *  subasm_set 

src.subtasks  :=  combine(subasm_set.stseq) 
subasm_set.priority  :=  src.priority 
subasm_set.mode  :=  async 

src  /E  pin  +  subasm_set 

src.subtasks  ;=  combine  (subasm_set.stseq) 
Subasm_set.priority  :=  src.priority 
subasm_set.mode  :=  sync 

pin  /E  component .  pinid 

pin.name  :=  component.lexicaivalueQ  + 
pinid. lexicalvaiueO 

Subasm_set  /E  ( subasm_list ) 

subasm_set.stseq  ;=  subasmjist.stseq 
subasmJist.priority  :=  subasm_set.priority 
subasmjist.mode  :=  subasm_set.mode 

Subasmjist  /E  subasm 

subasmjist.stseq  :=  seqadd(null,  subasm.subtasks) 
subasm.priority  :=  subasmJist.priority 
subasm.mode  :=  subasmjist.mode 

subasm_list  /E  subasm_list1  , 
subasm 

subasm_list.stseq  :=  seqadd(subasm_list1  .stseq,  sub¬ 
asm.subtasks) 

subasm.priority  :=  subasmJist.priority 
subasm.mode  :=  subasmjist.mode 

Subasm  /E  snk 

snk.epriority  :=  getpriority(subasm.priority,  snk.priority) 
subasm.subtasks  :=  prependsubtask(null,  snk.name, 
snk.epriority,  subasm.mode) 

Subasm  /E  snk  -  src_set 

snk.epriority  :=  getpriority(subasm.priority,  snk.priority) 
src_set.priority  :=  snk.epriority 
subasm.subtasks  :=  prependsubtask(src_set.subtasks, 
snk.name,  snk.epriority,  subasm.mode) 

snk/E  pin 

snk.priority  :=  pin.priority 
snk.name  :=  pin.name 

src_set  /E  ( src_list ) 

src_set.subtasks  :=  srcjist.subtasks 
src_list.priority  :=  src_set.priority 

src_list  /E  src 

src_list.subtasks  :=  src.subtasks 
src.priority  :=  srcjist.priority 

srcjist  /E  src_list1  ,  src 

src_list.subtasks  :=  prependseq(srcjist1  .subtasks, 
src.subtasks) 

src.priority  :=  src_list.priority 
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A.4.7  Types 

//  type  definitions 

t_seq  :  sequence  of  subtasks 

t_stseq  :  sequence  of  sequences  of  subtasks 

t_mode:  mode  of  activation  of  a  sink:  sync,  async 


A.4.8  Functions 

addtask(int  period,  t_seq  subtasks)  { 

adds  a  periodic  task  with  the  specified  period  and  subtasks  to  the 

analytic  assembly 

} 

t„seq  seqadd ( t_stseq  a,  t_seq  b)  { 
t_seq  r 
r  :=  a  +  <b> 
return  r 

} 

t_seq  prependseq ( t_seq  a,  t_seq  b)  { 
t_seq  r  : =  a 
r  :=  r  +  b 
return  r 

} 

t_seq  prependsubtask(t_seq  a,  char  name,  int  priority,  t_mode  mode)  { 
t_subtask  st 
St. name  :=  name 
st. priority  :=  priority 
St. mode  =  mode 
t_seq  b  =  <st>  +  a 
return  b 

} 

t_seq  combine (t_stseq  a)  { 
t_seq  b  : =  <> 
for  each  t_seq  c  in  a  { 
t_seq  t  :=  b 

//  combine  t  and  c  into  b 
b  :  =  <> 

while  t.size  >  0  &&  f irst ( t ) .mode  ~  sync  { 
b  :=  b  +  <first ( t) > 
t  :=  tail(t) 

} 

while  t.size  >  0  &&  c.size  >  0  { 

if  first (c) .priority  >  f irst ( t) .priority  { 
b  :=  b  +  <first{c)> 
c  :=  tail{c) 

}  else  { 

b  :=  b  +  <first{t)> 
t  :=  tail{t) 

) 

} 

if  t.size  >  0  { 
b  :=  b  +  t 

}  else  { 

b  :=  b  +  c 

} 

} 

return  b 

) 
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Appendix  B  X^ba  Empirical  Validation 


B.1  Introduction 

Validation  for  the  controller  PECT  consists  of  empirically  quantifying  the  accuracy  and  confi¬ 
dence  of  the  predictions  based  on  A,^ba-  "^^is  quantification  is  done  by  statistically  comparing 
predictions  with  actual  measurements  of  assembly  properties  and  establishing  confidence  in 
the  results,  which,  in  turn,  increases  confidence  in  the  analysis  model. 

This  appendix  describes  in  detail  the  experiment  to  empirically  validate 


B.2  Empirical  Validation 

Sections  B.2.1  -  B.2.6  correspond  to  steps  1  -  6  in  Figure  15  on  page  47. 


B.2.1  Define  Validation  Goal 

The  goal  for  the  controller  PECT  is  that  the  latency  for  a  job  within  the  hyper-period  can  be 
predicted  with  an  MRE  <  0.05  (5%)  with  a  confidence  level  y  =0.99  (99%).  A  minimum 
acceptable  p  =  0.80  (80%)  was  established  for  a  pass/fail  condition.  Latency  is  defined  as  the 
average  latency  of  each  job  in  each  period  of  the  assembly  hyper-period. 


B.2.2  Define  Measures 

B.2.2.1  Time  and  Pins 

Which  time  is  measured  and  how  is  fundamental  to  this  empirical  validation,  the  controller 
PECT,  and  Although  the  specific  measure  for  time  differs  between  the  property  for 
components  (Section  B.2.2.2)  and  assemblies  (Section  B.2.2.3),  the  units  of  time  are  the  same, 
and  the  same  apparatus  is  used. 

The  measurement  infrastructure  put  into  place  is  hosted  on  a  Microsoft  Windows  platform  and 
is  conformant  to  the  VenturCOM  RTX  clock  counter,  which  has  a  resolution  on  the  order  of 
<  l|is.^^  Time  recorded  by  the  measurement  infrastructure  using  this  clock  counter  is  in  milli¬ 
seconds  (ms)  with  eight  digits  of  precision. 
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A  time-stamped  event  is  generated  when  a  pin  is  entered  (activated)  or  exited  (deactivated). 
When  such  an  event  is  generated,  the  measurement  infrastructure  reads  the  clock  counter’s 
value  and  records  it  with  the  event.  Doing  so  provides  two  pieces  of  information — a  time- 
ordered  sequence  of  pin  events  and  a  means  of  determining  the  time  interval  between  any  two 
events. 


B.2.2.2  Component  Execution  Time 

When  a  pin  (source  or  sink)  is  entered,  it  is  considered  activated.  Activation  occurs  when 

•  sink  pin  enter  event.  The  component  is  in  receipt  of  a  message  or  request  (stimulus).  This 
is  the  start  of  a  reaction. 

•  source  pin  enter  event.  The  component  is  making  a  request  or  sending  a  message 
(response).  This  is  the  start  of  an  interaction. 

When  a  pin  (source  or  sink)  is  exited,  it  is  considered  deactivated.  Deactivation  occurs  after  a 


•  sink  pin  leave  event.  The  component  has  satisfied  the  request.  This  is  the  end  of  a  reaction. 

•  source  pin  leave  event.  The  component  has  completed  making  the  request  or  sending  the 
message.  This  is  the  end  of  an  interaction. 


The  above  time-recorded  events  define  measurement  points  when  measuring  individual  com¬ 
ponents  or  an  assembly  of  components.  For  example,  consider  the  component  shown  in  Figure 


37. 


R,  =  s-»r-^R,^ 


Figure  37:  Pin  Component  Enter  and  Leave  Events 

The  reaction  rule  for  this  simple  component  could  be  expanded  from  that  illustrated  in  Figure 
37  to  show  the  specific  enter  and  leave  events  asRs  =  s— >r— >r— >s— >  Rj,  where  x  represents 
the  specific  leave  event  for  a  given  sink  or  source  pin.  Figure  38  illustrates  six  time  measure¬ 
ments  on  a  component  based  on  enter  and  leave  events  for  sink  and  source  pins. 


34.  RtGetClockTimeO  reports  time  in  100  nanosecond  units. 
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Figure  38:  Time  Measurements  on  Component  Sink  and  Source  Reactions 

When  two  or  more  components  are  assembled,  the  number  of  possible  measurement  points 
(due  to  the  increase  in  events  generated  by  additional  components)  allows  additional  calcula¬ 
tions  to  be  made  across  component  interactions. 

For  ^abA’  total  execution  time  of  reactions  is  of  interest — specifically,  tj,  where  t4  =  0  for 
all  source  pins  in  the  reaction  (see  Figure  39). 


Figure  39:  Measuring  the  Execution  Time  of  a  Pin  Component 


B.2.2.3  Assembly  Latency 

An  interaction  between  components  exists  when  the  source  pin  of  one  component  is  connected 
to  the  sink  pin  of  another.  Further,  interactions  between  components  can  continue  as  a 

•  linear  interaction  (as  shown  in  Figure  40) 

•  multilinear  interaction  (as  shown  in  Figure  41) 

•  one-to-many  interaction  (as  shown  in  Figure  42) 

•  many-to-one  interaction  (as  shown  in  Figure  43) 

•  many-to-many  interaction  (as  shown  in  Figure  44) 
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Figure  40:  A  Simple  Linear  Interaction 


Figure  42:  A  Simple  One-to-Many  Interaction 
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Figure  43:  A  Simple  Many-to-One  Interaction 


Figure  44:  A  Simple  Many-to-Many  Interaction 

An  assembly  is  a  set  of  one  or  more  interactions.  A  task  is  defined  as  the  set  of  interactions 
starting  with  one  component’s  source  pin  and  terminating  at  another  component’s  sink  (or 
source)  pin.  Consider  the  small  assembly  depicted  in  Figure  40.  The  set  of  interactions  from 
Ci.r2  C3.S2  (i.e.,  {C].r2  C2.S2,  C2.r2  C3.S2})  is  considered  a  task.  A  job  is  the  scheduled 

periodic  execution  of  a  task;  a  task  has  one  or  more  jobs  in  a  hyper-period.  For  latency 
for  any  job  in  any  period  is  of  interest. 

Latency  for  a  job  is  measured  from  the  time  the  first  source  pin  is  activated  (Ci.r2  from  Figure 
40)  to  the  time  the  last  sink  pin  is  deactivated  (C3.S2),  as  illustrated  in  Figure  45. 


Figure  45:  Measuring  the  Latency  for  a  Simple  Assembly 
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B.2.3  Define  Sampling  Procedure 

The  overall  process  for  validating 

1 .  Predict  the  latency  of  the  longest  path  jobs  in  a  sample  of  assemblies. 

2.  Measure  the  actual  latencies  of  those  jobs. 

3.  Compare  predicted  and  actual  latencies. 

To  perform  the  steps  above,  we  needed  the  following: 

1.  a  population  of  components  (with  measured  execution  times) 

2.  a  representation  population  and  sufficient  quantity  of  assemblies  built  from  those  compo¬ 
nents 

The  components  and  their  assemblies  did  not  exist — a  situation  that  may  arise  frequently  in 
the  early  stages  of  PECT  development.  Thus,  we  were  forced  to  develop  synthetic  components 
and  assemblies. 


B.2.3.1  Component  Sample 

To  create  a  viable  population  of  components  to  choose  from,  15  classes  of  synthetic  compo¬ 
nents  were  created.  The  differentiation  between  each  class  was  strictly  controlled — specifi¬ 
cally  execution  time.  The  classes  were  named  sequentially  from  synthetics  through 
synthetic20,  where  the  numerical  postfix  indicated  the  number  of  milliseconds  for  which  the 
component  was  designed  to  execute.  The  calculation  that  was  performed  by  this  synthetic 
component  is  shown  in  Figure  46. 

Line.  1  for  (int  i  =  INT_MIN  /  765000;  i  <  0;  i++)  { 

Line.  2  for  (int  j  =  0;  j  <  execution;  j++)  { 

Line.  3  double  t  =  sqrt(abs(i))  *  sqrt (abs ( j ) ) ; 

Line.  4  ) 

} 

Figure  46:  Execution  Calculation  Made  by  the  Synthetic  Component 

The  variable  execution  in  Line  2  of  Figure  46  is  initialized  for  all  instances  for  each  class  to  a 
value  that  approximates  the  desired  execution  delay  (e.g.,  execution  =  Sms.  for  synthetics  and 
6ms.  for  synthetic6). 

The  interface  and  reaction  rules  for  each  synthetic  component  was  identical.  Each  synthetic 
component  had  one  synchronous  (sj)  and  one  asynchronous  (S2)  sink  pin,  as  well  as  one  syn¬ 
chronous  (rj)  and  one  asynchronous  (r2)  source  pin.  The  interface  for  each  pin  (named  signal) 
was  identical,  taking  no  in  or  in/out  arguments  and  having  no  return  value,  which  is  equiv- 
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alent  to  the  C-language  specification  void  signal  ( void) .  The  asynchronous  sink  pin 
(So)  was  handled  by  one  thread.  Finally,  the  reaction  rules  for  the  synthetic  component  were 
RS 1  =  s  [  — ^  r  j  — >  r  j  — ^  ro  — ^  T2  — ^  Sj  — >  RS 1  and  RS2  =  S2  — ^  r2  — ^  r2  — ^  r  j  — >  r  |  — ^  S2  — ^  RS2. 
The  reaction  rule  for  either  sink  pin  would  be  fired  only  after  the  internal  calculation  in  Figure 
46  was  performed. 

The  execution  time  (in  milliseconds)  for  each  reaction  was  measured  to  a  95%  confidence 
interval.  The  number  of  samples  was  chosen  to  achieve  that  target  precision.  In  most  cases, 
that  precision  was  achieved  before  30  samples.  For  each  reaction,  the  average  execution  time 
and  standard  deviation  were  recorded. 

In  addition  to  synthetic  components,  the  environment  Clock  component  was  developed.  This 
component  is  used  to  periodically  generate  a  signal  on  a  source  pin  for  a  specific  period  (e.g., 
every  200ms).  The  period  can  be  configured  prior  to  the  execution  of  any  instance  of  that 
class,  and  the  signal  can  be  generated  on  either  a  synchronous  or  asynchronous  source  pin. 

For  the  validation  of  it  was  only  necessary  to  measure  the  execution  time  of  synthetic 
components.  The  Clock  component  has  no  “internal”  calculation  and  is  used  as  the  stimulus 
for  various  jobs  within  the  assembly.  The  remainder  of  this  section  discusses  the  procedure  for 
measuring  synthetic  components. 

The  steps  for  sampling  the  execution  time  of  the  synthetic  component  are 

1 .  Create  a  test  Pin  component  assembly  for  each  synthetic  component  to  be  measured. 

2.  Execute  each  synthetic  component  through  the  runtime  environment  and  capture  its  mea¬ 
sured  execution  time. 

3.  Update  the  Pin  component  description  (registry)  for  each  component  tested  with  the  cap¬ 
tured  measures  from  the  runtime  environment. 

Each  step  is  described  in  detail  below. 

Step  1 :  Create  a  test  Pin  component  assembly  for  each  synthetic 
component  to  be  measured. 

This  step  is  done  using  a  series  of  MS-DOS  batch  (.BAT)  files.  The  createsinktest.bat  script  is 
used  to  generate  a  component  description  for  each  synthetic  component  class  having  an 
approximate  designed  execution  time  of  Sms  (i.e.,  synthetic5)  to  20ms  (i.e.,  synthetic20).  That 
script  calls  the  createonesinktest.bat  script  to  generate  a  single  component  description  for  the 
synthetic  component  enumerated  as  a  parameter.  The  output  from  this  initial  step  is  a  sequence 
of  synthetic  component  classes,  each  with  its  own  unique  execution  time  in  the  general  form 
shown  in  Figure  47. 
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Line . 

1  <?xinl  version= "  1 . 0 "  ?> 

Line . 

2 

<Asseinbly  xmlns= "  PinTekXML  .  xsd "  > 

Line . 

3 

<Components> 

Line , 

4 

<Component  name= " c 1 ockl "  type= " clock " > 

Line . 

5 

<Property  propId= "measurement "  value=" 

true"  /> 

Line. 

6 

<Property  propId= "period"  value=" 100 " 

/> 

Line . 

7 

</ Component > 

Line . 

8 

<Component  name= " taskl "  type= " synthet ic5 

"  > 

Line . 

9 

<Property  propId= "measurement "  value= " 

true"  /> 

Line . 

10 

<Property  propId= "priority"  value="10" 

/> 

Line . 

11 

<Property  propId="loadFactor"  value="5 

"  /> 

Line . 

12 

</Component> 

Line . 

13 

<Component  name=" task2 "  type="syntheticN 

”  > 

Line . 

14 

<Property  propId= "measurement "  value=" 

true"  /> 

Line. 

15 

<Property  propId= "priority"  value="15" 

/> 

Line . 

16 

<Property  propId= " loadFactor "  value= "N 

"  /> 

Line. 

17 

</ Component > 

Line . 

18 

< /Component s> 

Line . 

19 

<Connectors> 

Line. 

20 

<Connector> 

Line. 

21 

<Source  component="clockl "  pin="0"  /> 

Line. 

22 

<Sink  component=" taskl"  pin="l"  /> 

Line, 

23 

</Connector> 

Line. 

24 

<Connector> 

Line. 

25 

<Source  component^" taskl"  pin="2"  /> 

Line . 

26 

<Sink  component=" task2 "  pin="0"  /> 

Line. 

27 

</Connector> 

Line. 

28 

<Connector> 

Line . 

29 

<Source  component=" taskl"  pin="3"  /> 

Line. 

30 

<Sink  component=" task2 "  pin="l"  /> 

Line. 

31 

</Connector> 

Line . 

32 

</Connectors> 

Line. 

33 

</Assembly> 

Figure  47:  General  Assembly  Topology  Description  for  Benchmarking  Synthetic 

In  lines  13  and  16  of  Figure  47,  where  N  was  the  configured  execution  time  for  the  instances  of 
synthetic  components  being  tested.  N  for  this  validation  ranged  from  5  to  20,  inclusive.  Figure 
48  shows  the  general  topological  form  for  each  synthetic  component  tested. 


Figure  48:  General  Assembly  Topology  for  Benchmarking  Synthetic 
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Step  2:  Execute  each  synthetic  component  through  the  runtime 
environment  and  capture  its  measured  execution  time. 

The  synthetic  components  are  run  using  the  sinktestscript.bat  and  measure2.bat  scripts. 

These  scripts  invoke  the  Pin  runtime  environment  and  assembly  measurement  environment 
using  the  test  Pin  component  assemblies  built  in  the  previous  step.  Each  test  component 
assembly  is  run  twice:  once  for  the  synchronous  pin  of  the  synthetic  component  (task2.S| 
task2.si)  and  once  for  the  asynchronous  pin  of  the  same  synthetic  component  (task2.S9 
task2.S2).  The  test  assembly  is  run  until  it  is  stopped  by  the  assembly  measurement  environ¬ 
ment.  The  measurement  environment  will  cease  measuring  once  it  has  achieved  a  95%  confi¬ 
dence  interval  in  the  latency  of  the  measured  reaction.  In  all  cases,  the  95%  confidence 
interval  was  achieved  before  the  minimum  number  of  samples,  which  was  set  to  30. 


Step  3:  Update  the  Pin  component  description  (registry)  for  each 

component  tested  with  the  captured  measures  from  the  runtime 
environment. 

As  the  measurement  environment  measures  the  component  from  the  runtime  traces  emanated 
from  the  runtime  environment,  it  is  continuously  computing  the  average  elapsed  execution 
time  (in  milliseconds)  and  standard  deviation.  Once  the  95%  confidence  interval  is  met,  the 
measurement  environment  records  the  measures  for  the  component  and  exits  (thus  causing  the 
runtime  environment  to  cease).  The  form  of  the  results  for  each  recorded  assembly  is  shown  in 
Figure  49. 


Line . 

1 

<Component Properties 

Line . 

2 

naine="sinktest\sinktestlO  .xml  .cfg" 

Line . 

3 

pin='’l "  sainples="30 ’* 

Line . 

4 

avgElapsedTime= "13.39002667"  stdDevElapsedTime= " 0 . 182 646 ” /> 

Line . 

5 

<Coinponent  Properties 

Line . 

6 

name="sinktest\sinktestlO .xml .cfg" 

Line . 

7 

pin="2"  samples="30" 

Line . 

8 

avgElapsedTime=" 12.07455667"  stdDevElapsedTime=" 0 . 023759 " /> 

Figure  49:  Example  Measures  for  Component  Execution  Time 

In  Figure  49,  component  synthetic  10  has  an  average  execution  time  for  synchronous  pin  Sj  of 
13.4ms  with  a  standard  deviation  of  0,18ms  and  an  average  execution  time  for  asynchronous 
pin  S2  of  12.1ms  with  a  standard  deviation  of  0.02ms.  The  actual  execution  time  for  each  syn¬ 
thetic  component  class  is  captured  in  Table  16  on  page  112. 

The  measures  collected  by  the  measurement  environment  were  transferred  to  the  Pin  compo¬ 
nent  description  for  each  synthetic  component  class  tested  in  this  procedure,  essentially  estab¬ 
lishing  those  measures  as  the  “certified”  execution  time  properties  for  those  components, 
under  the  runtime  environment  described  in  Section  B.2.4.2. 
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B.2.3.2  Assembly  Sample 

Rather  than  arbitrarily  selecting  and  manually  building  assemblies  from  the  population  of 
components  (possibly  introducing  bias  into  the  selection  process),  assemblies  were  randomly 
selected  from  a  population  of  possible  assemblies  defined  by  assembly  variation  points.  The 
variation  points  serve  to  define  an  assembly  design  space  (a  n-dimension  coordinate  system) 
of  possible  assemblies.  For  example,  no  assembly  was  considered  for  had  more  than 

50  components.  The  following  variation  points  were  identified: 

•  number  of  clocks — the  total  number  of  Clock  components  discussed  above  that  are  used 
as  a  stimulus  to  other  components  within  an  assembly 

•  number  of  components — the  total  number  of  components,  in  this  case,  synthetic  compo¬ 
nents,  that  make  up  an  assembly 

•  number  of  connections  per  source  pin — the  maximum  number  of  connections  allowed  for 
a  single  component’s  source  pin  to  be  connected  to  other  components’  sink  pins 

•  minimum  load  factor — ^the  minimum  execution  time  for  a  component  in  the  assembly. 
This  number  could  range  from  5  to  20  and  had  to  be  less  than  or  equal  to  the  maximum 
load  factor. 

•  maximum  load  factor — the  maximum  execution  time  for  a  component  in  the  assembly. 
This  number  could  range  from  5  to  20. 

•  harmonic  period — determination  of  whether  the  clocks  in  the  system  have  the  same  period 

•  minimum  clock  period — the  minimum  clock  period  for  the  clocks  in  the  assembly.  This 
value  must  be  less  than  or  equal  to  the  maximum  clock  period. 

•  maximum  clock  period — the  maximum  clock  period  for  the  clocks  in  the  assembly 

•  mixed  connections — determination  of  whether  the  connections  between  the  components 
in  the  assembly  are  homogeneous  (all  synchronous  or  all  asynchronous)  or  heterogeneous 
(a  mix  of  synchronous  and  asynchronous) 

•  percent  blocking — percentage  of  the  total  number  of  synchronous  pins  used  in  the  assem¬ 
bly  that  must  be  blocked  or  mutexed 

Judgment  is  involved  in  defining  variation  points  and  their  possible  values. 

To  further  facilitate  the  generation  of  synthetic  assemblies,  a  set  of  tuples  was  defined  that 
bound  a  value  V  for  a  variation  point  (i.e,  <Vq,  V/,  ...,  V„'>).  Each  tuple  represented,  in  some 
way,  a  stereotypical  assembly.  Thirteen  such  tuples  were  defined,  again  using  judgment.  An 
example  tuple  is  shown  in  Figure  50;  the  complete  set  of  tuples  is  shown  in  Table  9. 
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Line . 

1 

<Description  name="l" 

Line. 

2 

nof Clocks= " 1 "  nof Component s= " 2 " 

Line . 

3 

nofConnectionsPerSrc  =  ''  1" 

Line . 

4 

minLoad= " 5 "  maxLoad= " 5 " 

Line . 

5 

harmonicPeriods= " true " 

Line , 

6 

minPeriod= "20 "  maxPeriod= " 100 " 

Line . 

7 

mixedConnect ions= " false " 

Line. 

8 

percentblocking="50"  /> 

Figure  50:  Variation  Points  for  One  Assembly  Design  Space 


Table  9:  Variation  Points  for  All  Assembly  Design  Spaces 


Assembly  Design 
Space  Tuple 

O 

o 

O 

o 

#  of  Components 

#  of  Connections 

Minimum  Load  (ms) 

Maximum  Load  (ms) 

Harmonic  Period 

Minimum  Period  (ms) 

Maximum  Period  (ms) 

Mixed  Connections 

%  Blocking 

1 

1 

2 

1 

5 

5 

Y 

20 

100 

N 

50 

2 

4 

50 

2 

5 

20 

Y 

500 

1000 

N 

50 

3 

1 

2 

1 

5 

5 

N 

20 

100 

N 

50 

4 

1 

2 

1 

5 

5 

Y 

20 

100 

Y 

50 

5 

1 

2 

1 

5 

5 

N 

20 

100 

Y 

50 

6 

2 

4 

1 

5 

10 

Y 

100 

500 

Y 

0 

7 

2 

8 

1 

5 

10 

N 

20 

100 

N 

100 

8 

2 

8 

1 

5 

10 

N 

20 

100 

Y 

0 

9 

4 

35 

1 

5 

5 

Y 

500 

2000 

Y 

100 

10 

2 

4 

1 

5 

5 

Y 

200 

400 

Y 

25 

11 

4 

15 

1 

5 

5 

Y 

200 

400 

Y 

25 

12 

2 

15 

3 

20 

20 

Y 

40 

1200 

Y 

75 

13 

6 

30 

1 

5 

5 

Y 

200 

400 

Y- 

25 

Each  tuple  (one  row  of  Table  9)  was  given  to  the  SEIAssemblyGeneratorController  tool  to 
randomly  select  an  assembly  from  the  design  space.  Each  generated  assembly  was  tested  for 
^ABA  well-formedness  per  the  rules  described  in  Appendix  A.  If  the  assembly  was  not  well 
formed,  it  was  rejected,  and  the  assembly  generator  would  randomly  select  another  assembly 
from  the  design  space.  This  process  repeated  until  a  well-formed  (i.e.,  analyzable)  assembly 
was  found,  or  the  assembly  generator  reached  an  upper-bound  search  heuristic  and  terminated, 
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For  the  version  of  the  assembly  generator  used  in  this  validation  exercise,  that  upper  bound 
was  set  to  look  at  150  random  assemblies  before  giving  up.^^ 

Although  our  approach  to  random  assembly  generation  worked  for  our  needs,  it  proved  to  be 
inefficient;  in  future  versions,  we  will  assume  that  each  interpretation  will  be  bundled  with  its 
own  assembly  generator. 

Figure  51  is  one  of  the  assemblies  generated  randomly  from  tuple  #1  (i.e.,  the  first  row  in 
Table  9). 


Line . 

1 

<?xinl  version="l .  0"  encoding=”utf-8"  standalone=:"yes'' ?> 

Line. 

2 

<Assembly  xmlns=”PinTekXML.xsd"> 

Line . 

3 

<Coinponents> 

Line . 

4 

<Component  name= " ClockO "  type= " clock " > 

Line. 

5 

<Property  propId= "period"  value="40"  /> 

Line. 

6 

</Component> 

Line . 

7 

<Component  name= "Cl "  type= " synthetics " > 

Line . 

8 

<Property  propId=" measurement"  value="true"  /> 

Line . 

9 

<Property  propId= "priority"  value="68"  /> 

Line . 

10 

<Property  propId=" loadFactor"  value="5"  /> 

Line . 

11 

<Property  propId= "blocking"  value="true"  /> 

Line . 

12 

< /Component > 

Line . 

13 

<Component  name= "C2 "  type= " synthetics " > 

Line . 

14 

<Property  propId= "measurement"  value="3"  /> 

Line . 

15 

<Property  propId="priority"  value="124"  /> 

Line, 

16 

<Property  propId=" loadFactor "  value="5"  /> 

Line. 

17 

<Property  propId=" blocking"  value=" false"  /> 

Line. 

18 

< /Component > 

Line . 

19 

< /Component s> 

Line. 

20 

<Connectors> 

Line . 

21 

<Connector> 

Line . 

22 

<Source  component = "ClockO"  pin="0"  /> 

Line . 

23 

<Sink  component="Cl"  pin="l"  /> 

Line . 

24 

</Connector> 

Line . 

25 

<Connector> 

Line . 

26 

<Source  component="Cl "  pin="3"  /> 

Line . 

27 

<Sink  component="C2 "  pin="l"  /> 

Line . 

28 

</Connector> 

Line . 

29 

</Connectors> 

Line . 

30 

</Assembly> 

Figure  5 1 :  Example  of  a  Randomly  Selected  Assembly 


35.  An  additional  search  heuristic  could  be  added  if  an  assembly  is  not  found  within  a  specified  period  of  time.  For  this  validation 
experiment,  most  assemblies  were  found  within  10  minutes.  Looking  for  longer  than  20  minutes  seemed  to  be  fruitless.  Per* 
haps  a  better  approach  would  be  to  construct  an  assembly  that  is  analyzable,  rather  than  selecting  assemblies  at  random. 
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Figure  52:  Topology  of  an  Example  of  a  Randomly  Selected  Assembly 

Given  an  analyzable  assembly,  the  tool  then  simulated  its  execution  considering  the  number  of 
jobs  found  within  it.  Each  job  would  be  traced,  and  latency  for  the  job  would  be  computed  via 
simulation  using  the  “certified”  execution  time  properties  for  each  pin  used  on  each  compo¬ 
nent  in  the  sequence  of  components  used  in  that  job  (see  Section  B. 2.5.1).  The  latency  com¬ 
puted  for  the  job  became  the  prediction  for  the  latency  of  that  job  for  that  assembly.  For  the 
assembly  shown  in  Figure  52,  the  calculated  prediction  (based  on  the  component  property  for 
pin  1  of  the  synthetic5  component  [see  Table  16  on  page  112])  is  shown  in  Table  10. 


Table  10:  Predicted  Latency  for  Synthetics’s  Job 


Task 

Job 

Predicted  Latency 
(ms) 

Standard  Deviation 
(+/-  ms) 

1 

1 

12.1573 

0.022722 

Task  #1  (from  Table  10)  refers  to  the  job  beginning  with  component  instance  clockO  in  the 
assembly.  Job  #1  is  the  “longest  path”  emanating  from  the  clock  that  can  be  predicted.  Since, 
in  this  assembly,  there  is  only  one  “path”  (i.e.,  clockO.rj  Cj.S2,  Ci.r2  C2.S2),  it  is,  by 
default,  the  “longest  path”  and  therefore  the  only  job — whence  one  prediction.  Predicted  latency 
is  the  prediction  for  the  average  number  of  milliseconds  it  will  take  to  complete  one  period  in 
the  hyper-period  for  the  job,  which  for  Task  #1,  Job  #1  is  11.75ms.  The  standard  deviation  is 
the  computed  standard  deviation  for  the  prediction  (in  milliseconds). 

With  a  prediction  for  each  job  in  each  defined  and  randomly  selected  assembly  (with  only  one 
being  shown  above),  each  assembly  was  run  through  the  runtime  environment  using  the 
assemblytestscript.bat  and  measureAssembly.bat  scripts.  These  scripts  invoke  the  Pin  runtime 
environment  and  assembly  measurement  environment  using  the  randomly  selected  component 
assemblies.  Each  random  component  assembly  is  run  once  for  each  job  found  in  the  assembly 
so  that  it  can  be  measured  independently  from  any  other  executing  jobs  in  the  assembly.  Each 
assembly  job  is  run  until  it  is  stopped  by  the  assembly  measurement  environment.  That  envi¬ 
ronment  will  cease  measuring  once  it  has  collected  N  samples  (which  in  this  case  was  30). 

The  measurement  environment  computes  continuously,  from  time-stamped  Pin  events,  the 
average  elapsed  time  (in  milliseconds),  and  the  standard  deviation  of  assembly  execution. 
Upon  completion  of  the  measurement  process,  the  measurement  environment  records  the  mea- 
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sures  for  the  assembly  and  exits.  The  form  of  the  results  for  each  recorded  assembly  is  shown 
in  Table  11. 

Table  11:  Observed  Average  Latency  for  an  Assembly 


Task 

Job 

Samples 

Average  Latency  (ms) 

Standard  Deviation  (+/-  ms) 

1 

1 

660 

12.180839727 

0.002032986 

Table  11  is  very  similar  to  the  table  described  above  for  recording  the  prediction.  The  differ¬ 
ences  are  that  after  successfully  measuring  a  job  for  an  assembly,  the  total  number  of  latency 
samples  observed  are  recorded  (in  this  case,  Samples  =  30),  and  the  computed  arithmetic  mean 
(i.e.,  Average  Latency  =  12.05ms)  is  calculated. 

Finally,  the  sampling  process  for  measuring  and  recording  the  actual  assembly  latencies  for 
each  job  in  all  the  selected  random  assemblies  was  repeated  22  times  resulting  in  22  complete 
sets  of  observations,  with  N  =  30  for  each  observation  (effectively,  N  =  22x30). 

The  actual  assembly  latencies  observed  for  each  job  are  captured  in  Table  17  on  page  113. 


B.2.4  Develop  Measurement  Infrastructure 

Clearly,  the  preceding  discussion  assumes  the  availability  of  a  measurement  infrastructure. 
The  infrastructure  described  here  was  developed  to  support  the  sampling  procedure  outlines  in 
Section  B.2.3.  We  expect  a  large  portion  of  this  infrastructure  to  be  reusable  by  other  property 
theories,  not  necessarily  time-based  ones. 


B.2.4.1  Measurement  Tool  Suite 

The  complete  measurement  infrastructure  required  to  obtain  measurement  data  on  the  compo¬ 
nents  and  assemblies  used  in  the  empirical  validation  is  discussed  briefly  below. 

The  overall  control  and  data  flow  for  the  tools  created  for  the  empirical  validation  is  shown  in 
Figure  53.  Component  execution  time  is  measured  by  placing  components  in  a  test  harness — 
essentially  an  assembly  occupied  only  by  the  component  (the  Composer  tool).  An  assembly 
generator  uses  variation  points  (the  tuples  described  earlier)  to  generate  well-formed  assem¬ 
blies  from  measured  components.  The  generated  assemblies  are  then  executed  in  an  instru¬ 
mented  runtime  that  records  actual  job  latencies.  This  data  is  then  compared  with  predictions 
to  produce  statistical  labels  for  X^ba- 
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Figure  53:  Measurement  Infrastructure  Tool  Workflow 

The  specific  tools  that  were  created  and  their  purposes  are  shown  in  Table  12. 

Table  12:  Measurement  Infrastructure  Tool  Suite 


Workflow  Element/Tool 

Purpose 

Component  repository 

reg.bat 

Registers  RTX  components  (.rtdils)  with  the  RTX  Runt¬ 
ime  executive 

*.rtdll 

RTX  components 

*.dll 

WIN32  components 

Composer 

createsinktest.bat 

Outer  control  loop,  iterating  from  5  to  20  times  as  argu¬ 
ments  for  the  component  assembly  generator  script 

createonesinktest.bat 

Component  assembly  generator  script,  creating  a  sim¬ 
ple  assembly  for  testing  a  single  component.  The  name 
of  the  component  is  passed  as  an  argument. 

XML2CFGsinktest.bat 

Outer  control  loop,  iterating  over  the  assemblies  gener¬ 
ated  by  the  component  assembly  generator  and  pass¬ 
ing  the  generated  XML  file  to  the  XML-to-CFG 
translator 
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Table  12:  Measurement  Infrastructure  Tool  Suite  (ConVd.) 


Workflow  Element/Tool 

Purpose 

Composer  (cont’d.) 

S  E  IT  ranslate  Assembly- 
fromXML.exe 

XML-to-CFG  translator  that  converts  an  assembly  gen¬ 
erated  by  the  component  assembly  generator  from  an 
XML  representation  to  a  CFG  representation  suitable 
for  consumption  by  the  RTX  Pin  Runtime 

Component  test  bench 

sinktestscript.bat 

Outer  control  loop,  iterating  over  the  assemblies  gener¬ 
ated  by  the  component  assembly  generator  and  pass¬ 
ing  the  converted  CFG  file  to  the  main  measurement 
script 

measure2.bat 

Main  measurement  script  overseeing  the  execution  of 
the  assembly  in  the  RTX  Pin  Runtime  and  measured  by 
the  RTX  Assembly  Measurement  tool.  When  the  RTX 
Assembly  Measurement  tool  is  complete,  the  script  kills 
the  RTX  Pin  Runtime  and  exits. 

results2.xml 

Component  measurements  (results)  of  actual  values 
captured  by  the  RTX  Assembly  Measurement  tool. 

Component  test  bench  OR  controller  assembly  test  bench 

Runtime.rtss 

RTX  Pin  Runtime.  Besides  executing  the  Pin  assembly, 
the  RTX  Pin  Runtime  generates  measurement  events 
that  are  consumed  by  the  RTX  Assembly  Measurement 
tool. 

assemblymsrmt.rtss 

RTX  Assembly  Measurement  tool.  Reads  measurement 
events  generated  by  the  RTX  Pin  Runtime  and  calcu¬ 
lates  the  job  execution  time  for  a  hyper-period  based  on 
command  line  arguments 

mywait.bat 

Used  to  pause  the  execution  of  the  main  measurement 
script  until  the  RTX  Pin  Runtime  has  been  signaled  by 
the  RTX  Assembly  Measurement  tool  that  it  is  com¬ 
plete.  The  mywait.bat  script  and  the  RTX  Assembly 
Measurement  tool  do  this  signaling  through  the  creation 
and  removal  of  a  file  in  the  file  system. 

wait.exe 

A  tool  designed  to  block  (wait)  for  N  seconds  using 
native  WIN32  calls  for  sleep()  (a  non-busy  wait) 

killruntime.rtss 

Used  to  signal  the  RTX  Pin  Runtime  to  suspend  execu¬ 
tion  of  the  assembly  and  exit 

rtx-hack.exe 

A  tool  that  is  used  to  aid  the  continuous,  non-human 
interactive  execution  of  the  RTX  Pin  Runtime.  A  bug  in 
the  RTX  executive  causes  the  generation  of  an  RTX 
exception  that  can  be  dismissed  only  by  pressing  a  but¬ 
ton  on  the  exception  dialog.  This  WIN32  application 
looks  for  that  dialog  and  dismisses  the  dialog  automati¬ 
cally. 
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Table  12:  Measurement  Infrastructure  Tool  Suite  (Cont’d.) 


Workflow  Element/Tool 

Purpose 

Component  registry 

components.xml 

Registration  information  about  components  avaiiable  for 
use  within  the  Pin  Runtime  environment.  This  fiie  con¬ 
tains  path  locations  to  the  components  in  the  fiie  system 
as  well  as  pin  configurations  and  measurement  data  for 
those  components. 

components.cfg 

RTX-Pin-Runtime-translated  version  of  the  XML  repre¬ 
sentation  of  the  same  name.  This  fiie  is  suitable  for  use 
with  the  RTX  Pin  Runtime. 

XML2CFGcomponents.bat 

Simple  one-line  script  for  running  the  XML-to-CFG 
translator  for  the  component  registry 

S  E  IT  ranslateCompo- 
nentsXMLexe 

XML-to-CFG  translator  that  converts  a  component  reg¬ 
istry  from  an  XML  representation  to  a  CFG  representa¬ 
tion  suitable  for  consumption  by  the  RTX  Pin  Runtime 

Controller  assembly  design  s 

pace  description 

ControllerValidationDescr.xml 

XML  file  that  describes  a  number  of  valid  design  spaces 
from  which  assemblies  can  be  selected  randomly 

Assembly  generator  and  pred 

ictor 

S  E 1  AssemblyGeneratorCon- 
troller.exe 

Outer  control  loop,  iterating  over  randomly  selected 
assemblies  from  a  predetermined  assembiy  design 
space.  The  selected  assembly  is  passed  to  the  Pin 
Runtime  simulator  for  rejection  (a  non-schedulable 
assembly)  or  prediction  (a  schedulable  assembly). 

AsyncGen.exe 

Pin  Runtime  simulator.  If  the  assembly  is  schedulable,  a 
prediction  is  made  for  each  job  within  the  assembly.  If  it 
is  not  schedulable,  the  assembly  is  rejected. 

SEL#_*.xml 

Assembly  that  is  generated  by  the  SEIAssemblyGener- 
atorController.exe  script  and  deemed  schedulable  by 
the  Pin  Runtime  simulator 

SEL#_*.cfg 

CFG  representation  of  the  XML  assembly  generated  by 
the  SEITranslateAssemblyfromXML.exe  script 

SEI_#_*.csv 

Prediction  made  for  each  job  in  the  assembly  by  the  Pin 
Runtime  simulator 

SEL#_Mpi 

“Longest  path”  information  for  each  job  within  an 
assembly 

XML2CFGassemblies.bat 

Outer  control  loop.  Iterating  over  the  assemblies  gener¬ 
ated  by  the  SEIAssemblyGeneratorController.exe  script 
assembly  generator  and  passing  the  generated  XML  file 
to  the  XML-to-CFG  translator 

SEITranslateAssembly- 

fromXML.exe 

XML-to-CFG  translator  that  converts  an  assembly  gen¬ 
erated  by  the  component  assembly  generator  from  an 
XML  representation  to  a  CFG  representation  suitable 
for  consumption  by  the  RTX  Pin  Runtime 
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Table  12:  Measurement  Infrastructure  Tool  Suite  (Cont’d.) 


Workflow  Element/Tool 

Purpose 

Assembly  generator  and  predictor  (cont’d.) 

sed 

Tool  used  to  patch  the  generated  CFG  file  due  to  a  bug 
in  the  Pin  Runtime  simulator 

.Controller  assembly  test  bench 

doemall.bat 

Outermost  control  loop  to  repeat  execution  of  all  the 
assemblies  generated  by  the  SEIAssemblyGenerator- 
Controller.exe  script  and  the  Pin  Runtime  simulator.  As 
configured,  this  script  will  repeat  the  run  of  the  entire  set 
of  assemblies  40  times. 

2assemblytestscript.bat 

Support  script  for  the  outermost  control  loop 

assemblymeasure.bat 

Inner  control  loop  to  manage  the  execution  of  all  gener¬ 
ated  assemblies 

measureAssembly.bat 

Main  assembly  measurement  script  overseeing  the 
execution  of  the  assembly  in  the  RTX  Pin  Runtime  and 
measured  by  the  RTX  Assembly  Measurement  tool. 
When  the  RTX  Assembly  Measurement  tool  is  com¬ 
plete,  the  script  kills  the  RTX  Pin  Runtime  and  exits. 

Controller  measurement  aggregation  and  basic  calculations 

pcfu_analysis_v2.xls 

Excel  spreadsheet  to  collect  all  the  measurements 
recorded  by  the  Pin  Measurement  tool  into  one  place. 
Performs  basic  descriptive  statistics  on  the  data  and 
aggregates  (such  as  Average,  MRE,  and  standard  devi¬ 
ation).  Other  sheets  within  the  Excel  workbook  include 
those  on  graph  generation  and  other  manual  aids  for 
analysis. 

Modulel  .gatherpredictionsO 

Excel.VBA  script  for  automating  the  inclusion  and 
aggregation  of  recorded  measurement  data 

Modulel  -paccCompareO 

Excel.VBA  script  for  automating  the  separation  of  actual 
latency  records  from  the  bulk  of  other  collected  data. 
Also  aids  in  analysis. 

Modulel  .paccCompareAVGO 

Excel.VBA  script  for  automating  the  separation  of 

Actual  MREs  calculated  from  the  bulk  of  other  collected 
data.  Also  aids  in  analysis. 

Controller  analysis  script 

pect1  .exa 

Script  used  for  automating  the  analysis  that  determines 
the  statistical  labels  from  the  experiment 

Stlnt.exe 

Tool  created  by  Hahn  and  Meeker  that  comes  complete 
with  a  rudimentary  statistical  scripting  language  [Hahn 
91] 
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B.2.4.2  Test  Environment 


Execution  time  properties  for  components,  as  well  as  assembly  job  latency,  are  directly 
affected  by  the  runtime  environment.  Such  properties  include  both  hardware  (e.g.,  processor 
speed)  and  software  (e.g.,  operating  system)  properties.  The  significant  hardware  and  software 
features  and  configurations  are  listed  in  Table  13  and  Table  14,  respectively. 


Table  13:  Hardware  Characteristics 


Item 

Value 

System  Manufacturer 

Dell  Computer  Corporation 

System  Model 

DIM4400 

Processor 

x86  Family  1 5  Model  1  Stepping  2 

Genuineintel  -1595  Mhz 

Physical  Memory 

1,047,856  KB 

184-pin  DIMMPC2100 

Virtual  Memory 

3,570,352  KB 

Hard  Disk  1 

WDC  WD400BB-75CAA0 

40  GB 

7200  RPM 

2  MB  Buffer 

8.9  ms  Average  Read  Seek  Time 

Table  14:  Software  Characteristics 


Item 

Value 

Operating  System  Name 

Microsoft  Windows  2000  Professional 

5.0.2195  Service  Pack  2  Build  2195 

Compiler 

Microsoft  Visual  C#  NET 

1.0  Build  3705 

Compiler 

Microsoft  Visual  Studio  C++ 

6.0  Service  Pack  5 

Real-Time  Extensions 

VenturCom  RTX 

5.1.1  Build  3517 

Additionally,  the  property  values  collected  can  be  affected  by  other  operating  and  user  pro¬ 
cesses  currently  running  on  the  machine.  All  opportunities  were  taken  to  “quiet”  the  runtime 
environment.  Table  15  lists  all  the  residual  processes  running  at  test  time. 
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Table  15:  Processes  (Running  Tasks) 


Name 

Version 

smss.exe 

5.00.2195.2901 

csrss.exe 

5.00.2195.2581 

winlogon.exe 

5.00.2195.2953 

services.exe 

5.00.2195.2780 

lsass.exe 

5.00.2195.2964 

svchost.exe 

5.00.2134.1 

spoolsv.exe 

5.00.2161.1 

svchost.exe 

5.00.2134.1 

mdm.exe 

7.00.9466 

regsvc.exe 

5.00.2195.2104 

mstask.exe 

4.71.2195.1 

winmgmt.exe 

1.50.1085.0029 

logon.scr 

5.00.2195.2104 

B.2.5  Collect  Validation  Data 

B.2.5.1  Component  Execution  Times 

Table  16  lists  the  actual  execution  times  recorded  for  each  synthetic  Pin  component. 
Table  16:  Recorded  Actual  Execution  Time  Per  Synthetic  Pin  Component 


Component 

Pin 

Samples 

Average  Execution 
(ms) 

Standard 
Deviation 
(+/-  ms) 

synthetic5 

si 

30 

6.86182667 

0.113103 

S2 

30 

6.08006000 

0.016712 

synthetic6 

s1 

30 

8.15998000 

. 

0.052514 

s2 

30 

7.28474333 

0.019437 

synthetic7 

s1 

30 

9.44960000 

0.109422 

s2 

30 

8.48425333 

0.019146 

synthetic8 

si 

30 

10.90788667 

0.236071 

s2 

30 

9.67879333 

0.023449 

synthetic9 

s1 

30 

12.22811333 

0.240077 

S2 

30 

10.87083667 

0.020344 

syntheticIO 

s1 

30 

13.39002667 

0.182646 

s2 

30 

12.07455667 

0.023759 
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Table  16:  Recorded  Actual  Execution  Time  Per  Synthetic  Pin  Component  (Cont’d.) 


Component 

Pin 

Samples 

Average  Execution 
(ms) 

Standard 
Deviation 
(+/-  ms) 

syntheticH 

s1 

30 

14.75889667 

0.200467 

s2 

30 

13.26221000 

0.025504 

synthetic12 

s1 

30 

16.13946333 

0.198329 

s2 

30 

14.45731333 

0.022695 

synthetic  13 

si 

30 

17.50676667 

0.283673 

s2 

30 

15.64764333 

0.027561 

synthetic  14 

s1 

30 

18.77334667 

0.253397 

s2 

30 

16.85212000 

0:026781 

syntheticIS 

s1 

30 

20.31739000 

0.340785 

s2 

30 

18.06251000 

0.028431 

syntheticie 

s1 

30 

21.42600667 

0.214585 

s2 

30 

19.24452333 

0.024071 

synthetic  17 

s1 

30 

22.79329667 

0.318561 

s2 

30 

20.44261000 

0.019831 

syntheticIS 

s1 

30 

24.41882667 

0.371608 

s2 

30 

21.63316667 

0.034281 

syntheticIS 

s1 

30 

25.44057000 

0.319016 

S2 

30 

22.83103333 

0.028673 

synthetic20 

s1 

30 

26.92379000 

0.018289 

s2 

30 

24.03515000 

0.027394 

B.2.5.2  Assembly  Latencies 

Table  17  lists  the  predicted  and  actual  latencies  recorded  for  each  job. 
Table  1 7:  Recorded  Predicted  and  Actual  Latencies  Per  Job 


Sample 

Assembly 

Task 

Job 

Predicted  Latency 
(ms) 

Average  Latency 
(ms) 

Average 

MRE  (+/-  ms) 
(AVGMRE) 

Standard  Deviation 

1 

0 

1 

1 

12.157434 

12.18083973 

0.002032986 

0.00324926 

2 

11 

1 

1 

152.203193 

142.799986 

0.065854237 

0.00246545 

3 

11 

2 

1 

223.729849 

212.6520431 

0.05209464 

0.001081566 
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Table  1 7:  Recorded  Predicted  and  Actual  Latencies  Per  Job  (ConVd,) 


Sample 

Assembly 

Task 

Job 

Predicted  Latency 
(ms) 

Average  Latency 
(ms) 

Average 

MRE  (+/-  ms) 
(AVGMRE) 

Standard  Deviation 

4 

11 

3 

1 

146.121677 

136.7693409 

0.068383718 

0.001940991 

5 

11 

3 

2 

108.057513 

105.05552 

0.028578573 

0.001880233 

6 

11 

3 

3 

108.057513 

105.0776989 

0.028361911 

0.002002774 

7 

11 

3 

4 

108.057513 

105.0631347 

0.028504362 

0.001974737 

8 

11 

4 

1 

295.342878 

288.4779173 

0.023797326 

0.000396185 

9 

11 

4 

2 

179.670542 

175.1159935 

0.026009175 

0.000673052 

m 

El 

4 

3 

179.670542 

175.1231314 

0.025967301 

0.000628656 

a 

D 

4 

4 

179.670542 

175.1215062 

0.025976785 

0.000595791 

■a 

■a 

1 

1 

36.480324 

36.37326986 

0.002992374 

0.004488773 

m 

2 

1 

12.160088 

12.14966386 

0.002546474 

0.002390286 

14 

14 

1 

1 

106.664623 

98.74003155 

0.080265901 

0.003149035 

15 

14 

2 

1 

140.99133 

128.7704897 

0.094915768 

0.00366488 

16 

14 

3 

1 

38.77888 

37.785479 

0.026291032 

0.000720778 

17 

14 

3 

2 

19.797934 

18.8435535 

0.050657309 

0.003273501 

18 

14 

4 

1 

72.319747 

68.42918259 

0.05685782 

0.001654267 

19 

14 

4 

2 

53.338801 

49.53426291 

0.076814768 

0.003101751 

O 

1 

1 

48.077234 

47.991087 

0.001796969 

0.001954304 

El 

■a 

1 

1 

6.077805 

6.090317524 

0.002716543 

0.004810158 

IQ 

■a 

1 

1 

6.077805 

6.087613143 

0.00254476 

0.004380116 

IQ 

El 

1 

1 

6.077805 

6.098547095 

0.003650455 

0.003595587 

El 

D 

1 

1 

593.455008 

634.2910765 

0.063112852 

0.034704991 

O 

n 

2 

1 

, 

995.947936 

995.237053 

0.000714403 

0.000352653 

n 

3 

1 

754.927901 

754.6399139 

0.000381712 

0.000307332 

o 

4 

1 

553.665296 

460.40705 

0.202556121 

0.000236622 

1 

1 

6.077805 

6.085927095 

0.003089996 

0.005467769 

IQI 

22 

1 

1 

24.050608 

23.97978685 

0.004547764 

0.007049814 

IQI 

22 

2 

1 

14.563308 

14.548268 

0.001903489 

0.001232084 

2 

2 

8.484954 

8.485028048 

0.00379372 

0.004132209 

1 

1 

72.517056 

72.18012165 

0.004674646 

0.002663979 

33 

23 

2 

1 

41.099661 

41.00429876 

0.002684425 

0.001086994 

34 

24 

1 

1 

65.423428 

62.56047962 

0.000855418 

0.001615869 

35 

24 

2 

1 

6.078576 

6.101276 

0.004783627 

0.00422444 

1 _ _ _ _ 
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Table  1 7:  Recorded  Predicted  and  Actual  Latencies  Per  Job  (ConVd.) 


Sample 

Assembly 

Task 

Job 

Predicted  Latency 
(ms) 

Average  Latency 
(ms) 

Average 

MRE  (+/-  ms) 
(AVGMRE) 

Standard  Deviation 

36 

25 

1 

1 

69.193544 

66.0317199 

0.047897211 

0.003892495 

37 

25 

2 

1 

16.965219 

16.94532629 

0.001287014 

0.00117919 

38 

27 

1 

1 

233.000651 

233.7746158 

0.063037527 

0.058178158 

39 

27 

2 

1 

304.571515 

279.4908092 

0.101757274 

0.128024805 

40 

27 

3 

1 

445.338419 

429.3696862 

0.03719116 

0.000247189 

41 

27 

3 

2 

140.766904 

137.2639743 

0.025519674 

0.00013175 

42 

27 

4 

1 

161.388063 

155.6482179 

0.036877181 

0.000394938 

43 

28 

1 

1 

12.156365 

12.10472462 

0.00780738 

0.013553423 

44 

28 

2 

1 

31.940761 

31.87197114 

0.002436647 

0.000894564 

45 

28 

2 

2 

25.86259 

25.797565 

0.002891515 

0.001465898 

46 

2 

1 

1 

12.157^34 

12.19095824 

0.003216353 

0.003132189 

47 

30 

1 

1 

207.820552 

172.1396488 

0.212449269 

0.077072795 

48 

30 

1 

2 

70.788164 

67.99884624 

0.041023116 

0.001819544 

49 

30 

2 

1 

187.238652 

192.6022863 

0.059528467 

0.026309187 

50 

30 

3 

1 

166.674934 

135.7480596 

0.227828031 

0.00179305 

51 

30 

4 

1 

146.123674 

117.9770064 

0.238577791 

0.000535831 

52 

31 

1 

1 

1087.456557 

1032.780649 

0.052941272 

0.000932037 

53 

31 

2 

1 

1036.499941 

788.6200498 

0.314321675 

0.000919638 

54 

3 

1 

1 

6.077805 

6.092341667 

0.003273897 

0.003919068 

55 

5 

1 

1 

12.908483 

12.92091581 

0.001746366 

0.002360845 

56 

7 

1 

1 

20.554505 

20.53108629 

0.002418703 

0.002413149 

57 

7 

1 

2 

20.554505 

20.50707924 

0.003266289 

0.002385814 

58 

7 

1 

3 

20.554505 

20.51884838 

0.002771204 

0.002421099 

59 

7 

1 

4 

20.554505 

20.51021686 

0.003258122 

0.002549417 

60 

7 

1 

5 

20.554505 

20.51467581 

0.002937428 

0.002512625 

61 

7 

2 

1 

48.552451 

48.50180686 

0.001356255 

0.000789963 

62 

8 

1 

1 

58.062533 

58.01301033 

0.000994399 

0.00057472 

63 

8 

2 

1 

37.516312 

37.45211567 

0.002025215 

0.000869905 

64 

9 

1 

1 

64.903463 

64.92833071 

0.002105629 

0.001895588 

65 

9 

2 

1 

32.412077 

32.43519214 

0.001379683 

0.001714629 
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B.2.6  Analyze  Results 

The  goal  of  the  analysis  study  for  empirically  validating  the  property  theory  (in  Appendix  A) 
was  to  support  or  refute  the  hypothesis  that  would  predict  the  latency  for  any  job  within 
the  hyper-period  with  an  MRE  <  0.05  with  a  confidence  level  y  =0.99.  A  minimum  acceptable 
p  =  80%  was  established  for  a  pass/fail  condition. 


B.2.6.1  Computations 

Each  time  a  job  was  executed  during  the  sampling  procedure,  the  actual  latency  and  standard 
deviation  was  recorded.  Recall  that  this  was  done  22  times.  Therefore,  our  measurement  envi¬ 
ronment  recorded  22  average  latencies  for  each  job  where  each  recorded  average  latency  could 
be  compared  against  the  original  prediction  for  that  job  (see  Section  B.2.5.2). 


The  MRE  for  each  recorded  and  observed  average  latency  was  calculated  for  job  using  this 
equation: 


MRE 


Predicted- Actual 
Actual 


This  resulted  in  having  22  MREs  computed  for  each  job  executed.  From  those  MREs,  the 
arithmetic  mean  and  standard  deviation  (AVGMRE)  of  each  job  was  computed.  The  result  of 
that  computation  is  found  in  Table  17. 


B.2.6.2  Descriptive  Statistics 

Table  18  includes  some  basic  descriptive  statistics  for  the  assembly  measures  recorded  in 
Table  17. 

Table  18:  Descriptive  Statistics  for  Job  Assembly  Measurements 


Part  of  Interval 

Value 

Samples  (N) 

65 

Mean  MRE  (mAVGMRE) 

0.039612786 

Standard  Deviation  (s) 

0.064798 

Spearman  rank  correlation  of  Predicted 
Latency  and  Average  Latency® 

0.997 

p-Value  <  0.001 

a.  A  high  correlation  (closer  to  -1 .0  or  1 .0)  is  an  indication  of  correlation  between  the 
predicted  latency  and  average  latency.  P-value  is  the  probability  that  any  such 
correlation  is  a  coincidence.  For  a  small  p-value,  reject  the  hypothesis  that  the  cor¬ 
relation  is  a  coincidence.  Spearman's  correlation  is  a  non-parametric  correlation 
and  does  not  make  assumptions  about  the  data  (i.e.,  normality). 
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B.2.6.3  Normality  Test 

The  Shapiro-Wilk  normality  test  was  used  to  support  (or  refute)  the  hypothesis  that  the  popula¬ 
tions  of  job  AVGMREs  was  normal.  Depending  on  the  result,  different  forms  of  intervals  must 
be  used.  The  results  from  the  Shapiro-Wilk  normality  test  are  shown  in  Table  19. 

Table  19:  Results  from  the  Shapiro-Wilk  Normality  Test  on  AVGMRE 


W  statistic 

p-value 

0.6266 

1.383e-11 

Since  the  p-value  is  less  than  the  generally  accepted  critical  p-value  of  0.05,  the  hypothesis 
that  the  MREs  for  the  jobs  are  normally  distributed  was  rejected. 

Next,  a  logarithmic  transformation  was  applied  to  the  AVGMREs  with  the  intent  to  make  them 
normally  distributed.  That  is,  each  AVGMRE  in  the  population  was  transformed  using 
LOG( AVGMRE)  and  a  new  population  was  created.  Again,  the  Shapiro-Wilk  normality  test 
also  refuted  that  the  new,  transformed  population  was  normal  because  the  p-value,  although 
much  improved,  was  still  smaller  than  the  critical  p-value  of  0.05. 

Table  20:  Results  from  the  Shapiro-Wilk  Normality  Test  on  LOG(AVGMRE) 


^statistic 

p-value 

0.9217 

5.282e-04 

Similarly,  a  Box-Cox  transformation  also  failed  to  produce  a  normal  distribution  (varying 
{  5.0,  0.5, 0.3, 0.1, 0.01  }3^).  Histograms  of  the  AVGMRE  and  LOG(AVGMRE)  popula¬ 
tion  are  shown  in  Figures  54  and  55  below. 


36.  As  the  X-values  approach  0,  the  transformation  approaches  the  logarithmic  transformation  performed  earlier.  This  is  not  sur¬ 
prising,  because  the  Box-Cox  transformation  forA-value  0  equals  LOG(). 
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000  0.05  0  10  0.15  0  20  0.25  0  30  0  35 

AVGMRE 

Figure  54:  Histogram  of  AVGMRE  Job 


-3.5  -3.0  -2.5  -2.0  -1.5  -1.0  -0.5 

LOG(AVGMRE) 

Figure  55:  Histogram  of  LOG(AVGMRE)  Job 

Given  that  the  population  of  the  AVGMRE  job  data  is  not  normal,  normal  distribution  statisti¬ 
cal  intervals  cannot  be  used.  Therefore,  distribution-free  statistical  intervals  had  to  be  used  to 
compute  the  tolerance  and  confidence  intervals. 
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B.2.6.4  One-Tail  Distribution-Free  Statistical  Intervals 


The  upside  to  using  distribution-free  statistical  intervals  is  that  there  are  no  assumptions  about 
the  data’s  normality.  Given  that  the  population  of  the  AVGMREs  recorded  is  not  normal,  this 
upside  is  an  incentive  for  using  distribution-free  intervals.  However,  the  downside  is  that  it  is 
difficult  to  achieve,  precisely,  the  desired  confidence  intervals.  Further,  the  intervals  that  are 
determined  tend  to  be  longer  than  those  derived  from  normal  distributions.  One  reason  for  this 
difference  is  that  such  statistical  methods  do  not  require  parameters  to  the  formulae  and  table 
lookups  (such  as  standard  deviation)  that  are  used  to  determine  either  tolerance  or  confidence 
intervals  for  distribution-free  populations^^  [Hahn  91]. 

The  goal  for  this  analysis  is  to  support  or  reject  the  hypothesis  that  the  predict  the 

latency  of  a  job  to  have  an  MRE  <  0.05.  This  is  the  upper  bound  for  an  MRE  interval  repre¬ 
senting  situations  where  predictions  would  be  no  worse  than  the  ^iMRE  for  a  job.  Notice  that 
the  lower  bound  for  an  MRE  interval  would  represent  situations  where  the  predictions  would 
be  no  better  than  the  ^iMRE  for  a  job.  Thus,  knowing  both  ends  of  the  interval  (a  two-tail 
interval)  is  not  the  goal  of  this  analysis  study,  as  only  the  upper-bound  interval  (a  one-tail 
interval)  is  of  interest. 

Stint,  a  tool  that  performs  a  number  of  different  kinds  of  statistical  intervals  (including  distri¬ 
bution-free  intervals),  was  used  to  determine  the  tolerance  interval  for  the  population  of  job 
AVGMREs  [Meeker  93].  The  first  calculation  performed  is  shown  in  Figure  56  where 

•  p=0 . 80  is  the  proportion  of  the  percentile  of  the  population  to  compute  the  interval. 

•  kside=2  is  the  upper  tolerance  bound. 

•  conlev=  0 . 9  9  is  the  desired  confidence  level. 


Line. 
Line. 
Line. 
Line, 
least 
Line . 
Line. 
Line, 
Line. 
Line. 
Line, 
Line. 


1  stint>  dfti  p=0.80  kside=2,  conlev=0.99 

2  finding  a  nonparametric  99.0%  upper  confidence  bound 

3  for  the  80.0  percentile. 

4  this  is  also  a  99.0%  upper  tolerance  bound  to  exceed  at 

5  80.0%  of  the  population. 

6 

7  the  interval  (bound)  is  based  on  65  observations. 

8  the  desired  upper  bound  is  x(  60) . 

9  the  actual  confidence  level  is=  0*9942 

10 

11  ordered  observation  number  60  is  0.10 


Figure  56:  Distribution-Free  Calculation  Using  Stint 


37.  Distribution-free  statistical  intervals  are  also  referred  to  as  non-parametric  statistical  intervals. 
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The  result  of  this  initial  calculation  is  that  our  hypothesis  for  achieving  an  MRE  <  0.05  was  not 
supported.  The  upper-bound  tolerance  we  could  expect  to  achieve  based  on  this  population 
(N=65,  line  7  above)  was,  at  best,  0.10  (or  10%  MRE.  line  11  above)  for  80%  of  the  popula¬ 
tion,  with  an  actual  confidence  level  of  99.42%  (line  9  above). 

Table  21:  Results  from  the  Initial  Calculation  for  a  One-Sided  Distribution-Free 
Tolerance  Interval 


Part  of  Interval 

Value 

N  =  65 

sample  size 

Y  =  0.9942 

confidence  level 

p  =  0.80 

proportion 

^MRE  =  -0.04 

mean  MRE 

UB  =  10% 

upper  bound 

Based  on  trial  and  error  (iterating  though  different  values  of  p),  it  was  determined  that  an 
upper-bound  tolerance  of  5%  MRE  could  be  achieved  for  62%  of  the  population  with  an  actual 
confidence  of  99.22%.^* 

These  results  were  somewhat  surprising.  Why  were  they  so  far  off  in  that  we  could  only 
achieve  10%  MRE  for  our  target  population?  It  was  time  to  look  at  the  data. 


B.2.6.5  Revisiting  the  Data 

Referring  back  to  the  histograms  in  Figures  54  and  55,  on  page  118,  there  was  a  noticeable 
skewing  of  the  data,  and  the  histograms  did  not  provide  much  insight.  Next,  upon  inspecting 
the  data  closer  in  Table  17,  we  immediately  noticed  (on  the  whole)  that  much  of  the  measure¬ 
ments  were  very  stable  (a  <  0.10)  and  fell  into  two  general  regions  (see  Figure  57); 

1.  Region  1 — samples  whose  calculated  MRE  was  within  our  goal  (MRE  <  0.05) 

2.  Region  2 — samples  whose  calculated  MRE  was  outside  our  goal  (MRE  >  0.05) 


38.  dfti  p=0.62,  kside=2,  and  conlev=0.99. 
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Figure  57:  Summary  of  the  AVGMRE  Job  Grouped  by  MRE 
From  this  perspective  of  the  data,  two  hypotheses  were  formulated: 

1.  Hypothesis  1  (Region  1  only):  Removing  the  systematic  error  (indicative  of  a  possible 
error  in  the  ^aba  property  theory,  its  interpretation,  or  the  runtime)  would  improve  the 
confidence  and  tolerance  intervals,  allowing  us  to  achieve  our  analytic  goal. 

2.  Hypothesis  2  (All  Regions):  The  systematic  errors  in  the  population  could  not  be  found  or 
corrected,  and  the  confidence  and  tolerance  intervals  calculated  in  Section  B.2.6.4  would 
stand. 

To  test  hypothesis  1,  the  original  data  set  of  65  observations  recorded  in  Table  17  was  reduced 
by  those  observations  represented  as  Region  2  in  Figure  57.  As  a  result,  the  data  set  for 
hypothesis  1  AV GMRE  consisted  of  47  observations.  (Those  observations  in  Region  2  were 
removed.)  The  descriptive  statistics  for  hypothesis  1  are  shown  in  Table  22.  (Hypothesis  2  is 
also  repeated  here  for  the  reader’s  convenience.) 


Table  22:  Descriptive  Statistics  for  Hypothesis  Data  Sets 


Statistic 

Hypothesis  1  AVGMRE 

Hypothesis  2  AVGMRE 

Samples  (N) 

47 

65 

Mean  MRE  (mAVGMRE) 

0.010487 

0.039612786 

Standard  Deviation  (s) 

0.013375 

0.064798 

Spearman  rank  correlation  of  Pre¬ 
dicted  Latency  and  Average  Latency 

0.997 

p-Value  <  0.001 

0.997 

p- Value  <  0.001 
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Using  the  Stint  tool  again,  the  one-sided,  distribution-free  tolerance  intervals  were  computed 
for  the  Region  1  data  set  (see  Table  23).  From  these  calculations,  the  upper-bound  tolerance 
interval  improved  substantially.  The  data  set  of  AVGMREs  for  hypothesis  1  (Region  1),  using 
a  one-sided,  distribution-free  tolerance  interval,  improved  to  4%  MRE  for  80%  of  the  popula¬ 
tion  with  a  confidence  of  99.07%. 


Table  23:  Hypothesis  Calculations  for  a  One-Sided,  Distribution- Free  Tolerance 
Interval 


Part  of 
Interval 

Hypothesis  1  AVGMRE 

Hypothesis  2  AVGMRE 

N 

47 

65 

Y 

0.9907 

0.9942 

P 

0.80 

0.80 

Pmre 

-0.01 

-0.04 

UB 

4% 

10% 

Next,  the  Region  1  data  set  was  tested  for  normality  and  failed  the  Shapiro- Wilk  test.  A  loga¬ 
rithmic  transformation  of  the  Region  1  data  set  also  failed  that  test.  We  would  have  to  live  with 
the  distribution-free  analysis  for  now. 

These  results  are  encouraging.  That  is,  if  both  the  systematic  errors  can  be  found  and  solved 
for  the  hypothesis  1)  and  future  results  follow  the  representative  population  of 

AVGMRE  (having  minimal  errors),  we  can  expect  to  achieve  an  MRE  <  5%  for  80%  of  the 
predictions  with  a  confidence  of  99%.  But  that  still  has  to  be  proven. 


B.2.7  Final  Statistical  Labels 

Table  24  is  a  compilation  of  the  data  from  the  previous  sections  organized  according  to  the  two 
hypotheses  derived  from  revisiting  that  data  (see  Section  B. 2.6.5  on  page  120).  Hypothesis  2 
corresponds  to  the  results  of  the  actual  validation.  All  calculations  assume  a  distribution-free 
.  tolerance  interval. 


Table  24:  Final  Statistical  Labels  for  All  Hypotheses 


Part  of  Interval 

Hypothesis  1 

Hypothesis  2 

sample  size  (N) 

47 

65 

_ _ _ 

confidence  level  (y) 

0.9907 

0.9942 

proportion  (p) 

0.80 

0.80 

m©3n  MRE  (^mre) 

-0.01 

-0.04 

upper  bound  (UB) 

4% 

10% 
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B.3  Conclusions 


The  goal  for  the  property  theory  that  the  latency  for  a  job  within  the  hyper-period  can  be 
predicted  with  an  MRE  <  0.05  with  a  confidence  level  y  =0.99  for  at  least  80%  of  the  popula¬ 
tion  was  not  supported  and  was  thereby  rejected. 

It  appeared  that  the  first  75%  of  the  job  MRE  predictions  (47  of  65,  the  optimistic  observa¬ 
tions)  were  represented  well  by  the  which  was  encouraging.  The  remainder  of  those 
predictions  appeared  to  be  systematic  errors  that  warrant  further  investigation  either  wrongly 
accounted  for  in  A,^ba  o*"  represented  at  all.  It  is  also  feasible  that  either  the  runtime  or 
measurement  environment,  or  both,  could  be  at  fault.  In  any  event,  further  co-refinement  of 
A-aba  i®  required. 


B.4  Stint  Program 

The  Stint  program  contains  the  following  code; 


Line . 

12 

batch 

Line . 

13 

c 

Line . 

14 

c  #  Actual 

AVGMRE  (Region 

L  1)  dataset 

Line . 

15 

c 

Line . 

16 

read  47  observations 

Line . 

17 

0.002032986, 

0.028578573, 

0.028361911, 

0.028504362, 

Line . 

18 

0.023797326, 

0.026009175, 

0.025967301, 

0.025976785, 

Line . 

19 

0.002992374, 

0.002546474, 

0.026291032, 

0.001796969, 

Line . 

20 

0.002716543, 

0.00254476, 

0.003650455, 

0.000714403, 

Line . 

21 

0.000381712, 

0.003089996, 

0.004547764, 

0.001903489, 

Line . 

22 

0.00379372, 

0.004674646, 

0.002684425, 

0.000855418, 

Line . 

23 

0.004783627, 

0.047897211, 

0.001287014, 

0.03719116, 

Line . 

24 

0.025519674, 

0.036877181, 

0.00780738, 

0.002436647, 

Line . 

25 

0.002891515, 

0.003216353, 

0.041023116, 

0.003273897, 

Line . 

26 

0.001746366, 

0.002418703, 

0.003266289, 

0.002771204, 

Line . 

27 

0.003258122, 

0.002937428, 

0.001356255, 

0.000994399, 

Line . 

28 

0.002025215, 

0.002105629, 

0.001379683 

Line . 

29 

c 

Line . 

30 

c  #  Calculate  mean  and  stdev 

Line. 

31 

c 

Line. 

32 

cndata 

Line . 

33 

c 

Line . 

34 

c  #  distribution  free  one 

i^tail  tolerance  interval 

Line . 

35 

c 

Line . 

36 

dfti  p=0.80 

kside=2 ,  conlev=0 . 99 

Line . 

37 

c 

Line . 

38 

c  #  Actual 

AVGMRE  (Region 

.  1  +  Region 

2 )  dataset 

Line . 

39 

c 

Line. 

40 

read  65  observations 

Line. 

41 

0.002032986, 

0.065854237, 

0.05209464, 

0.068383718, 

Line . 

42 

0.028578573, 

0.028361911, 

0.028504362, 

0.023797326, 
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Line . 
Line . 
Line . 
Line . 
Line. 
Line . 
Line . 
Line . 
Line . 
Line. 
Line. 
Line . 
Line . 
Line . 
Line . 
Line . 
Line. 
Line. 
Line . 
Line . 
Line. 
Line . 
Line . 
Line . 


43 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 

55 

56 

57 

58 

59 

60 
61 
62 

63 

64 

65 

66 


0.026009175, 

0.002546474, 

0.050657309, 

0.002716543, 

0.000714403, 

0.001903489, 

0.000855418, 

0.063037527, 

0.00780738, 

0.041023116, 

0.001746366, 

0.003258122, 

0.002025215, 

0.212449269, 

0.101757274 

c 


0.025967301, 

0.080265901, 

0.05685782, 

0.00254476, 

0.000381712, 

0.00379372, 

0.004783627, 

0.03719116, 

0.002436647, 

0.059528467, 

0.002418703, 

0.002937428, 

0.002105629, 

0.227828031, 


0,025976785, 

0.094915768, 

0.076814768, 

0.003650455, 

0.003089996, 

0.004674646, 

0.047897211, 

0.025519674, 

0.002891515, 

0.052941272, 

0.003266289, 

0.001356255, 

0.001379683, 

0.238577791, 


0.002992374, 

0.026291032, 

0.001796969, 

0.063112852, 

0.004547764, 

0.002684425, 

0.001287014, 

0.036877181, 

0.003216353, 

0.003273897, 

0.002771204, 

0.000994399, 

0.202556121, 

0.314321675, 


c  #  Calculate  mean  and  stdev 
c 

cndata 


c 

c  #  distribution  free  one-tail  tolerance  interval 
c 

dfti  p=0.80  kside=2,  conlev=0.99 
stop 
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Appendix  C  NuSMV  Model  of  CSWI 


This  appendix  contains  the  NuSMV  model  for  the  CSWI  component  described  in  Chapter  7. 

MODULE  main 
IVAR 

input  :  {_opsel_on,  _opsel_off,  _oppos_open,  _oppos_close, 
sbosel,  sbopos,  none}; 

VAR 

output  :  {_sbosel_on,  _sbosel_off,  _sbopos_open, 
_sbopos_close,  opsel,  oppos,  none}; 

state  :  {waiting,  selecting,  selected,  opening,  closing, 
deselecting,  unselecting} ; 

ASSIGN 

init{ input)  :=  none; 

next (input)  is  unconstrained/non-deterministic 

ini t (output)  :=  none; 
next (output)  := 
case 

state  =  waiting 
state  =  waiting 
state  =  selecting 
state  =  selected 
state  =  selected 

state  =  selected 

state  =  selected 
_sbopos_close ; 

state  =  opening 
state  =  closing 

state  =  deselecting  &  input  =  sbosel  :  oppos 

state  =  unselecting  &  input  =  sbosel  :  opsel 

1  :  none ; 


&  input  = 
Sc  input  = 
Sc  input  = 
Sc  input  = 
Sc  input  = 


_opsel_on 
__opsel__of  f 
sbosel 
_opsel_on 
_opsel_of f 


:  _sbosel_on 
;  _sbosel_off 
:  opsel 
:  _sbosel_on 
:  sbosel  off 


Sc  input  =  _oppos_open  :  _sbopos_open 


Sc  input  =  _oppos_close  : 


Sc  input  =  sbopos 
Sc  input  =  sbopos 


_sbosel_of  f 
.sbosel  of f 
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esac  ; 

init (state)  :=  waiting; 
next (state)  := 
case 


State 

= 

waiting 

Sc 

input 

_opsel_on 

selecting 

state 

= 

waiting 

Sc 

input 

= 

__opsel_of  f 

unselecting 

state 

selecting 

Sc 

input 

= 

sbosel 

selected 

state 

= 

selected 

Sc 

input 

= 

_opsel_on 

selecting 

state 

= 

selected 

Sc 

input 

= 

_opsel_of f 

unselecting 

state 

selected 

Sc 

input 

= 

_oppos_open 

opening 

state 

= 

selected 

Sc 

input 

= 

__oppos_close 

closing 

state 

= 

opening 

Sc 

input 

:= 

sbopos 

deselecting 

state 

= 

closing 

Sc 

input 

= 

sbopos 

deselecting 

state 

= 

deselecting 

Sc 

input 

= 

sbosel 

waiting 

state 

= 

unselecting 

Sc 

input 

sbosel 

waiting 

1  :  state; 
esac  ; 

SPEC  EF( state  =  waiting) 

SPEC  EF( state  =  selecting) 

SPEC  EF (state  =  selected) 

SPEC  EF( state  =  opening) 

SPEC  EF( state  =  closing) 

SPEC  EF( state  =  deselecting) 

SPEC  EF( state  =  unselecting) 

SPEC  AG( (state  =  waiting)  &  (input  =  _opsel_on)  ->  AX(output  = 
_sbosel_on) ) 

SPEC  AG ( (state  =  selected)  &  (input  =  _opsel_off)  ->  AX(output 
=  _sbosel_off ) ) 

SPEC  AG( (state  =  selected)  &  (input  =  _oppos_open)  ->  AX(output 
=  _sbopos_open) ) 

SPEC  AG( (state  =  selected)  &  (input  =  _oppos_close)  ->  AX(out- 
put  =  _sbopos_close) ) 

SPEC  !E[! (output  =  _sbosel_on)  U  (output  =  _sbopos_open) ] 

--  the.  following  claim  is  expected  to  fail 
SPEC  AG (input  =  _opsel_on  ->  AG  output  =  none) 
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Appendix  D  Switch  Schematic 


The  schematic  for  the  SEI  switch  is  shown  in  its  entirety  in  Figure  58.  Figures  59  through  62 
show  the  same  schematic  in  larger  scale  split  into  four  quadrants  for  easier  viewing.  To  get  a 
copy  of  the  original  schematic,  contact  Kurt  Wallnau  via  email  at  kcw@sei.cmu.edu. 


Figure  58:  SEI  Switch:  Full  Schematic 
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REVnaONS 


Figure  59:  SEI  Switch:  Upper  Right  Quadrant 
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Figure  60:  SEI  Switch:  Lower  Right  Quadrant 
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Acronym  List 


ABB 

Asea  Brown  Boveri,  Ltd. 

ABB/CRC 

Asea  Brown  Boveri,  Ltd./Corporate  Research  Center 

AlP 

Aspect  Integration  Program 

API 

application  program  interface 

COTS 

commercial  off-the-shelf 

CPU 

central  processing  unit 

CTL 

computational  tree  logic 

DoD 

Department  of  Defense 

EMS 

energy  management  system 

FIFO 

first  in,  first  out 

HP 

hyper-period 

lEC 

International  Electrotechnical  Commission 

lED 

intelligent  electronic  device 

I/O 

input/output 

LCM 

least  conunon  multiple 

LIFO 

last  in,  first  out 

LN 

logical  node 

LS-SDE 

language-specific  software  development  environment 
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LTL 

linear  temporal  logic 

MRE 

magnitude  of  relative  error 

MS 

milliseconds 

NS 

nanoseconds 

OLE 

object  linking  and  embedding 

OPC 

OLE  (object  linking  and  embedding)  for  process  control 

OS 

operating  system 

PACC 

predictable  assembly  of  certifiable  components 

PC 

physical  connection 

PD 

physical  device 

PECT 

prediction-enabled  component  technology 

RMA 

rate  monotonic  analysis 

RUP 

Rational  Unified  Process 

SAS 

substation  automation  system 

SCADA 

supervisory  control  and  data  acquisition 

SDE 

software  development  environment 

SEI 

Software  Engineering  Institute 

TBD 

to  be  decided 

UDP 

Universal  Datagram  Protocol 

UML 

Unified  Modeling  Language 

XML 

Extensible  Markup  Language 

134 


CMU/SEI-2002-TR-031 


Bibliography 


URLs  valid  as  of  the  publication  date  of  this  document 


[Aho  87] 

[Allen  97] 

[Bachmann  00] 


[Ball  02] 


[Barnett  01] 


[Bass  98] 


[Buskins  02] 


Aho,  A.;  Sethi,  R.;  &  Ullman,  J.  Compilers:  Principles,  Tech¬ 
niques,  and  Tools.  Reading,  MA;  Addison-Wesley,  1987. 

Allen,  R.  “A  Formal  Approach  to  Software  Architecture.”  PhD 
diss.  (CMU-CS-97-144),  Carnegie  Mellon  University,  May  1997. 

Bachmann,  F.;  Bass,  L.;  Buhman,  C.;  Comella-Dorda,  S.;  Long, 
F.;  Robert,  J.;  Seacord,  R.;  &  Wallnau,  K.  Volume  II:  Technical 
Concepts  of  Component-Based  Software  Engineering,  2nd  Edi¬ 
tion  (CMU/SEI-2000-TR-008,  ADA379930).  Pittsburgh,  PA: 
Software  Engineering  Institute,  Carnegie  Mellon  University, 
2000.  <http://www.sei.cmu.edu/publications/documents 
/00.reports/00tr008.html>. 

Ball,  T.  &  Rajamani,  S.  K.  “Automatically  Validating  Temporal 
Safety  Properties  of  Interfaces,”  103-122.  Proceedings  of  the 
Workshop  on  Model  Checking  of  Software,  LNCS  2057.  Toronto, 
Ontario,  Canada,  May  19-20,  2001.  New  York,  NY:  Springer- 
Verlag,  2002. 

Barnett,  M.  &  Schulte,  W.  “Spying  on  Components:  A  Runtime 
Verification  Technique,”  7-13.  Proceedings  of  the  Workshop  of 
Specification  and  Verification  of  Component-Based  Systems 
(SAVCBS)  at  OOPSLA.  Tampa,  Florida,  October  14,  2001.  New 
York,  NY:  Association  for  Computing  Machinery,  2001. 
<http://www.cs.iastate.edu/~leavens/SAVCBS/papers-2001 
/bamett-schulte.pdf>  or  <http://research.microsoft.com 
/foundations/savcbs.pdf>. 

Bass,  L.;  Clements,  R;  &  Kazman,  R.  Software  Architecture  in 
Practice.  Boston,  MA:  Addison-Wesley,  1998. 

Buskins,  V.  Social  Networks  and  Trust.  Boston,  MA:  Kluwer 
Academic  Publishers,  2002. 


CMU/SEt-2002-TR-031 


135 


[CDIF  97] 


[Cimatti  00] 


[Clarke  99] 


[Clements  02a] 


[Clements  02b] 


[Davies  93] 


[Fenton  97] 


[Gardiner  00] 


[Garlan  93] 


[Giannakopoulou 

99] 


Electronic  Industries  Association.  CDIF  CASE  Data  Interchange 
Format — Overview,  <http://www.eigroup.org/cdif 
/electronic-extracts/OV-extract.pdf>  (1997). 

Cimatti,  A.;  Clarke,  E.;  Giunchiglia,  F.;  &  Roveri,  M.  “NuSMV: 
A  New  Symbolic  Model  Verifier.”  International  Journal  on  Soft¬ 
ware  Tools  for  Technology  Transition  2, 4  (April  2000):  410-425. 
<http://nusmv.irst.itc.it/NuSMV/papers/sttt_j/html/index.html>. 

Clarke,  E.;  Grumberg,  O.;  &  Peled,  D.  A.  Mode!  Checking.  Cam¬ 
bridge,  MA:  MIT  Press,  Massachusetts  Institute  of  Technology, 
1999. 

Clements,  P.  &  Northrop,  L.  Software  Product  Lines:  Practices 
and  Patterns.  Boston,  MA:  Addison-Wesley,  2002. 

Clements,  R;  Bachmann,  F.;  Bass,  L.;  Garlan,  D.;  Ivers,  J.;  Little, 
R.;  Nord,  R.;  &  Stafford,  J.  Documenting  Software  Architectures: 
Views  and  Beyond.  Boston,  MA:  Addison-Wesley,  2002. 

Davies,  J.  Specification  and  Proof  in  Real-Time  CSP.  New  York, 
NY:  Cambridge  University  Press,  1993. 

Fenton,  N.  E.  &  Pleeger,  S.  L.  Software  Metrics  A  Rigorous  and 
Practical  Approach  (ISBN  1-85032-275-9).  Boston,  MA:  PWS 
Publishing  Company,  1997. 

Gardiner,  R;  Goldsmith,  M.;  Hulance,  J.;  Jackson,  D.;  Roscoe,  B.; 
&  Scattergood,  B.  Failures-Divergence  Refinement:  FDR2  User 
Manual  Oxford,  England:  Formal  Systems  (Europe)  Ltd.,  2000. 

Garlan,  D.  &  Shaw,  M.  “An  Introduction  to  Software  Architec¬ 
ture,  1-39.  Proceedings  of  the  4th  International  Conference  on 
Software  Engineering  and  Knowledge  Engineering,  Advances  in 
Software  Engineering  and  Knowledge  Engineering.  Capri,  Italy, 
June  15-20,  1992.  River  Edge,  NY;  World  Scientific  Rublishing 
Company,  1993. 

Giannakopoulou,  D.  “Model  Checking  for  Concurrent  Software 
Architectures.”  PhD  diss..  Imperial  College  of  Science,  Technol¬ 
ogy,  and  Medicine,  University  of  London,  January,  1999. 


136 


CMU/SEI-2002-TR-031 


[Gluch  02] 

[Gorton  02] 

[Hahn  91] 

[Havel  und  00] 

[Hissam  02] 

[Hoare  85] 

[Hoizman  97] 

[lEC  02] 

[Ivers  02] 


Gluch,  D.;  Comella-Dorda,  S.;  Hudak,  J.;  Lewis,  G;  Walker,  J.; 
Weinstock,  C.;  &  Zubrow,  D.  Model-Based  Verification:  An 
Engineering  Practice  (CMU/SEI-2002-TR-021,  ADA407768). 
Pittsburgh,  PA:  Software  Engineering  Institute,  Carnegie  Mellon 
University,  2002.  <http://www.sei.cmu.edu/publications 
/documents/02.reports/02tr02 1  .html>. 

Gorton,  I.  &  Liu,  A.  “Software  Component  Quality  Assessment 
in  Practice:  Successes  and  Practical  Impediments,”  555-558.  Pro¬ 
ceedings  of  the  24th  International  Conference  on  Software  Engi¬ 
neering.  Orlando,  Florida,  May  19-25,  2002.  New  York,  NY: 
Association  for  Computing  Machinery,  2002. 

Hahn,  G  &  Meeker,  W.  Statistical  Intervals:  A  Guide  for  Practi¬ 
tioners.  New  York,  NY:  John  Wiley  &  Sons,  Inc.,  1991. 

Havelund,  K.  &  Pressburger,  T.  “Model  Checking  Java  Programs 
Using  Java  PathFinder.”  International  Journal  on  Software  Tools 
for  Technology  Transfer  (STTT)  2,  4  (April  2000):  366-381. 

Hissam,  S.;  Moreno,  G;  Stafford,  J.;  &  Wallnau,  K.  “Packaging 
Predictable  Assembly,”  108-124.  Proceedings  of  the  1st  ACM/ 
IFIP  Working  Conference  on  Component  Deployment  (Lecture 
Notes  in  Computer  Science  Volume  2370).  Berlin,  Germany,  June 
20-21,  2002.  Berlin,  Germany:  Springer  Verlag,  2002. 

Hoare,  C.  A.  R.  Communicating  Sequential  Processes.  Engle¬ 
wood  Cliffs,  NJ:  Prentice  Hall  International,  1985. 

Hoizman,  G  J.  ‘The  Model  Checker  Spin.”  IEEE  Transactions 
on  Software  Engineering  23, 5  (May  1997):  279-295. 

International  Electrotechnical  Commission.  Communications 
Networks  and  Systems  in  Substations,  Working  Draft  for  Interna¬ 
tional  Standard  lEC  61850-1. .10.  Geneva,  Switzerland:  Interna¬ 
tional  Electrotechnical  Commission,  January,  2002. 

Ivers,  J.;  Sinha,  N.;  &  Wallnau,  K.  A  Basis  for  Composition  Lan¬ 
guage  CL  (CMU/SEI-2002-TN-026,  ADA407797).  Pittsburgh, 
PA:  Software  Engineering  Institute,  Carnegie  Mellon  University, 
2002.  <http://www.sei.cmu.edu/publications/documents 
/02.reports/02tn026.html>. 


CMU/SEI-2002-TR-031 


137 


[Kazman  00] 


[Klein  93] 


[Kruchten  00] 


[Lamport  77] 


[Li  02] 


[Magee  93] 


[Magee  99] 


[Mason  02] 


[Meeker  93] 


Kazman.  R.;  Klein,  M.;  &  Clements,  R  ATAM:  Method  for  Archi¬ 
tecture  Evaiiiation  (CMU/SE1-2000-TR-004.  ADA382629). 
Pittsburgh,  PA:  Software  Engineering  Institute.  Carnegie  Mellon 
University,  2000.  <http;//www.sei.cmu.edu/pubIications 
/documents/00.reports/00tr004.html>. 

Klein,  M.;  Ralya,  H.;  Pollack,  B.;  &  Obenza,  R.  A  Practitioner's 
Handbook  for  Real-Time  Analysis:  Guide  to  Rate  Monotonic 
Analysis  for  Real-Time  Systems.  Boston,  MA:  Kluwer  Academic 
Press,  1993. 

Krutchen,  P.  The  Rational  Unified  Process  An  Introduction,  2nd 
edition.  Boston,  MA:  Addison-Wesley,  2000. 

Lamport,  L.  “Proving  the  Correctness  of  Multiprocess  Pro¬ 
grams.”  IEEE  Transactions  on  Software  Engineering  SE  3,  2 
(March  1997):  125-143. 

Li,  P.  L.;  Shaw,  M.;  Stolarick,  K.;  &  Wallnau,  K.  The  Potential 
for  Synergy  Between  Certification  and  Insurance. 
<http://www.sei.cmu.edu/staff/kcw/icsr02.pdf>  (2002). 

Magee,  J.;  Dulay,  N.;  &  Kramer,  J.  “Structured  Parallel  and  Dis¬ 
tributed  Programs.”  Software  Engineering  Journal  8,  2  (March 
1993):  73-82. 

Magee,  J.  &  Kramer,  J.  Concurrency  State  Models  &  Java  Pro¬ 
grams.  New  York,  NY:  Wiley,  1999. 

Mason,  D.  “Probabilistic  Analysis  for  Component  Reliability 
Composition.”  Proceedings  of  the  5th  International  Workshop  on 
Component-Based  Software  Engineering:  Benchmarks  for  Pre¬ 
dictable  Assembly,  in  conjunction  with  the  24th  International 
Conference  on  Software  Engineering  (ICSE2002).  Orlando,  Flor¬ 
ida,  May  19-20,  2002.  <http://www.sei.cmu.edu/pacc/CBSE5 
/Mason-cbse5-fmal  .pdf>. 

Meeker,  W.  &  Chow,  I.  Stint— A  Computer  Program  for  Comput¬ 
ing  Statistical  Intervals,  <http://www.public.iastate.edu 
/~wqmeeker/StInt/StInt.pdf>  (1993). 


138 


CMU/SEI-2002-TR-031 


[Meyer  98] 


[Milner  89] 

[Milner  99] 

[Moreno  02] 


[Naumovich  00] 


[Piakosh  99] 


[Pnueli  85] 


[Preiss  01] 


[Prior  62] 


Meyer,  B.;  Mingins,  C.;  &  Schmidt,  H.  “Providing  Trusted  Com¬ 
ponents  to  the  Industry.”  Computer  (IEEE)  31,  5  (May  1998): 
104-105. 

Milner,  R.  Communication  and  Concurrency.  New  York,  NY: 
Prentice  Hall,  1989. 

Milner,  R.  Communicating  and  Mobile  Systems:  The  n-Calculus. 
New  York,  NY:  Cambridge  University  Press,  May  1999. 

Moreno,  G;  Hissam,  S.;  &  Wallnau,  K.  “Statistical  Models  for 
Empirical  Component  Properties  and  Assembly-Level  Property 
Predictions:  Toward  Standard  Labeling.”  Proceedings  of  the  5th 
International  Workshop  on  Component-Based  Software  Engi¬ 
neering,  in  conjunction  with  the  24th  International  Conference  on 
Software  Engineering  (ICSE2002).  Orlando,  Florida,  May  19-20, 
2002.  <http://www.sei.cmu.edu/pacc/CBSE5 
/Moreno-cbse5-final.pdf>. 

Naumovich,  G  &  Clarke,  L.  “Classifying  Properties:  An  Alterna¬ 
tive  to  the  Safety-Liveness  Classification,”  159-168.  Proceedings 
of  the  Eighth  International  Symposium  on  the  Foundations  of 
Software  Engineering.  San  Diego,  California,  November  8-10, 
2000.  New  York,  NY:  Association  for  Computing  Machinery, 
2000. 

Piakosh,  D.;  Smith,  D.;  &  Wallnau,  K.  Builder’s  Guide  for  Water- 
Beans  Components  (CMU/SEI-99-TR-024,  ADA373154).  Pitts¬ 
burgh,  PA:  Software  Engineering  Institute,  Carnegie  Mellon 
University,  1999.  <http://www.sei.cmu.edu/publications 
/documents/99.reports/99tr024/99tr024abstract.html>. 

Pnueli,  A.  &  Harel  D.  On  the  Development  of  Reactive  Systems 
(Technical  Report  CS85-02).  Rehovot,  Israel:  Weizmann  Institute 
of  Science,  Department  of  Computer  Science,  1985. 

Preiss,  O.  &  Wegmann,  A.  Towards  a  Composition  Model  Prob¬ 
lem  Based  on  1EC61850.  <http://www.sei.cmu.edu/pacc 
/CBSE4_papers/PreissWegmann-CBSE4-4.pdf>  (2001). 

Prior,  Arthur  N.  Formal  Logic,  2nd  edition.  Oxford,  UK:  Claren¬ 
don  Press,  1962. 


CMU/SEI-2002-TR-031 


139 


[Rabinovich  00] 


[Reed  88] 


[Roscoe  98] 


[Rumbaugh  00] 


[Schubert  99] 


[Schwetman  78] 


[Sharygina  02] 

[Stafford  01] 


[Stafford  02] 


[Szyperski  97] 


Rabinovich,  S.  G.  Measurement  Errors  and  Uncertainties,  2nd 
edition  (ISBN  0-387-98835-1).  New  York,  NY:  Springer- Verlag, 
2000. 

Reed,  G  M.  “A  Uniform  Mathematical  Theory  for  Real-Time 
Distributed  Computing.”  PhD  diss.,  Oxford  University,  1988. 

Roscoe,  A.  W.  The  Theory  and  Practice  of  Concurrency.  New 
York,  NY;  Prentice  Hall,  1998. 

Rumbaugh,  J.;  Jacobson,  I.;  &  Booch,  G  The  Unified  Modeling 
Language  Reference  Manual.  Reading,  MA:  Addison-Wesley 
Longman,  2000. 

Schubert,  P.  &  Ginsburg,  M.  “Virtual  Communities  of  Transac¬ 
tion:  The  Role  of  Personalization  in  Electronic  Commerce,”  646- 
663. 12th  International  Bled  Electronic  Commerce  Conference: 
“Global  Networked  Organizations.  ”  Bled,  Slovenia,  June  7-9, 
1999.  Bled,  Slovenia:  Modema  Organizacija,  1999. 

Schwetman,  H.  “Hybrid  Simulation  Models  of  Computer  Sys¬ 
tems.”  Communications  of  the  ACM  21,  9  (September  1978): 
718-723. 

Sharygina,  N.  “Model  Checking  of  Software  Control  Systems.” 
PhD  diss..  The  University  of  Texas  at  Austin,  2002. 

Stafford,  J.  &  Wallnau,  K.  Is  Third-Party  Certification  Neces¬ 
sary?  <http://www.sei.cmu.edu/pacc/CBSE4_papers 
/StafrordWallnau-CBSE4-22.pdf>(2001). 

Stafford,  J.  &  McGregor,  J.  D.  “Issues  in  Predicting  the  Reliabil¬ 
ity  of  Composed  Components.”  Proceedings  of  the  5th  Interna¬ 
tional  Workshop  on  Component-Based  Software  Engineering,  in 
conjunction  with  the  24th  International  Conference  on  Software 
Engineering  (ICSE2002).  Orlando,  Florida,  May  19-20,  2002. 
<http://www.sei.cmu.edu/pacc/CBSE5 
/StaffordMcGregor-cbseS  .pdf>. 

Szyperski,  C.  Component  Software:  Beyond  Object  Oriented  Pro¬ 
gramming.  Reading,  MA:  Addison-Wesley,  1997. 


140 


CMU/SEI-2002-TR-031 


[Taylor  94] 

[Then  02] 

[van  Glabbeek  01] 

[van  Ommering  02] 

[Walinau  02] 

[Walpole  89] 

[Woodcock  97] 


Taylor,  B.  &  Kuyatt,  C.  Guidelines  for  Evaluating  and  Expressing 
the  Uncertainty  of  NIST  Measurement  Results  (microform,  NIST 
Technical  Note  1297).  Gaithersburg,  MD;  U.S.  Department  of 
Commerce,  Technology  Administration,  National  Institute  of 
Standards  and  Technology,  1994. 

Thiel,  S.  &  Hein,  A.  “Modeling  and  Using  Product  Line  Variabil¬ 
ity  in  Automotive  Systems.”  IEEE  Software,  Special  Issue  on 
Software  Product  Lines  19,  4  (July/August  2002):  66-72. 

van  Glabbeek,  R.  Part  1,  Ch.  1,  ‘The  Linear  TimerBranching 
Time  Spectrum  1.  The  Semantics  of  Concrete,  Sequential  Pro¬ 
cesses,”  3-99.  Handbook  of  Process  Algebra  (ISBN  0-444- 
82830-3).  New  York,  NY:  Elsevier,  2001. 
<http://adam.wins.uva.nl/~alban/Handbook-free>. 

van  Ommering,  R.  Part  5,  Ch.  12,  ‘The  Koala  Component 
Model,”  223-236.  Building  Reliable  Component-Based  Software 
Systems.  Boston,  MA:  Artech  House,  2002. 

Walinau,  K.;  Hissam,  S.;  &  Seacord,  R.  Building  Systems  from 
Commercial  Components.  Boston,  MA:  Addison-Wesley,  2002. 

Walpole,  R.  E.  &  Myers,  R.  H.  Probability  and  Statistics  for 
Engineers  and  Scientists.  New  York,  NY:  MacMillan  Publishing 
Company,  1989. 

Woodcock,  J.  &  Davies,  J.  Using  Z:  Specification,  Refinement, 
and  Proof.  New  York,  NY:  Prentice  Hall,  1997. 


CMU/SEI-2002-TR-031 


141 


142 


CMU/SE1-2002-TR-031 


REPORT  DOCUMENTATION  PAGE 


Form  Approved  ' 
OMB  No.  0704-0188 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering 
and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of 
information,  including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  information  Operations  and  Reports,  1215  Jefferson  Davis  Highway.  Suite 
1204,  Arlington.  VA  22202-4302.  and  to  the  Office  of  Management  and  Budget,  Paperwork  Reduction  Project  (0704-01 88).  Washington,  DC  20503. 


10.  AGENCY  USE  ONLY  (leave  blank) 


13.  TITLE  AND  SUBTITLE 


11.  REPORT  DATE 


September  2002 


Predictable  Assembly  of  Substation  Automation  Systems;  An  Experiment 
Report 

AUTHOR(S) 

Scott  Hissam,  John  Hudak,  James  Ivers,  Mark  Klein,  Magnus  Larsson, 
Gabriel  Moreno,  Linda  Northrop,  Daniel  Plakosh,  Judith  Stafford,  Kurt 
Wallnau,  William  Wood 

m  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESSfES) 

Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

1 8.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

HQ  ESC/XPK 
5  Eglin  Street 

Hanscom  AFB,  MA  01731-2116 

20  SUPPLEMENTARY  NOTES 


1 2.  REPORT  TYPE  AND  DATES  COVERED 

Final 

14^  FUNDING  NUMBERS 

C  — F19628-00-C-0003 


17.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 

CMU/SEI-2002-TR-031 

To  SPONSORING/MONITORING 
AGENCY  REPORT  NUMBER 

ESC-TR-2002-031 


12.a  DISTRIBUTION/AVAILABILITY  STATEMENT 

Unclassified/Unlimited,  DTIC,  NTIS 

To  ABSTRACT  (maximum  200  words) 


|12.b  DISTRIBUTION  CODE 


The  Predictable  Assembly  from  Certifiable  Components  (PACC)  Initiative  at  the  Software  Engineering 
Institute  (SEI®*^)  is  developing  methods  and  technologies  for  predictable  assembly.  A  software  development 
activity  that  builds  systems  from  components  is  predictable  if  the  runtime  behavior  of  an  assembly  of 
components  can  be  predicted  from  known  properties  of  components  and  their  patterns  of  interactions 
(connections),  and  if  these  predictions  can  be  objectively  validated.  A  component  is  certifiable  if  these  known 
properties  can  be  obtained  or  validated  by  independent  third  parties.  The  SEI’s  technical  approach  to  PACC 
rests  on  prediction-enabled  component  technology  (PECT).  At  the  highest  level,  PECT  is  a  scheme  for 
systematic  and  repeatable  integration  of  software  component  technology,  software  architecture  technology, 
and  design  analysis  and  verification  technology.  This  report  describes  the  results  of  an  exploratory  PECT 
prototype  for  substation  automation,  an  application  area  in  the  domain  of  power  generation,  transmission, 
and  management.  This  report  focuses  primarily  on  the  methodological  aspects  of  PECT;  the  prototype  itself 
was  only  a  means  to  expose  and  illustrate  the  PECT  method. 


14.  SUBJECT  TERMS 


MSl  NUMBER  OF  PAGES 


component-based  software,  predictable  assembly,  component  technology,  1 64 

certification,  component  framework,  CBSE,  prediction-enabled  component  Ta  price  code 
technology 

SECURITY  CLASSIFICATION  1 1 8.  SECURITY  CLASSIFICATION  1 1 9.  SECURITY  20  LIMITATION  OF  ABSTRACT 

OF  REPORT  OF  THIS  PAGE  CLASSIFICATION 

UNCLASSIFIED  UNCLASSIFIED  of  abstract  yL 

UNCLASSIFIED 

NSN  7540-01  -280-5500  Standard  Form  298  (Rev.  Z-89) 

Prescribed  by  ANSI  Std.  239-1 0 


